Project_4_Multiple_View_Modeling/index.html at master · COMP5421/Project_4_Multiple_View_Modeling · GitHub

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
<html>
<head>
<title>COMP5421 Multiple View Modeling</title>
<link href='http://fonts.googleapis.com/css?family=Nunito:300|Crimson+Text|Droid+Sans+Mono' rel='stylesheet' type='text/css'>
<link rel="stylesheet" title="Default" href="styles/github.css">
<script src="http://ajax.googleapis.com/ajax/libs/jquery/1.3.2/jquery.min.js"></script>

<link rel="stylesheet" href="highlighting/styles/default.css">
<script src="highlighting/highlight.pack.js"></script>

<style type="text/css">
body {
	margin: 0px;
	width: 100%;
	font-family: 'Crimson Text', serif;
	font-size: 20px;
	background: #fcfcfc;
}
h1 {
	font-family: 'Nunito', sans-serif;
	font-weight: normal;
	font-size: 28px;
	margin: 25px 0px 0px 0px;
	text-transform: lowercase;
}
h2 {
	font-family: 'Nunito', sans-serif;
	font-weight: normal;
	font-size: 32px;
	margin: 15px 0px 35px 0px;
	color: #333;
	word-spacing: 3px;
}
h3 {
	font-family: 'Nunito', sans-serif;
	font-weight: normal;
	font-size: 26px;
	margin: 10px 0px 0px 0px;
	color: #333;
	word-spacing: 2px;
}
h4 {
	font-family: 'Nunito', sans-serif;
	font-weight: normal;
	font-size: 22px;
	margin: 10px 0px 10px 0px;
	color: #333;
	word-spacing: 2px;
}
h5 {
	font-family: 'Nunito', sans-serif;
	font-weight: normal;
	font-size: 18px;
	margin: 10px 0px 10px 0px;
	color: #111;
	word-spacing: 2px;
}
p, li {
	color: #444;
}
a {
	color: #C7EDCC;
}
.container {
	margin: 0px auto 0px auto;
	width: 1160px;
}
#header {
	background: #333;
	width: 100%;
}
#headersub {
	color: #ccc;
	width: 960px;
	margin: 0px auto 0px auto;
	padding: 20px 0px 20px 0px;
}
.chart {
	width: 480px;
}
.lol {
	font-size: 16px;
	color: #888;
	font-style: italic;
}
.sep {
	height: 1px;
	width: 100%;
	background: #999;
	margin: 20px 0px 20px 0px;
}
.footer{
	font-size: 16px;
}
.latex {
	width: 100%;
}
.latex img {
	display: block;
	margin: 0px auto 0px auto;
}
pre {
	font-family: 'Droid Sans Mono';
	font-size: 14px;
}
table td {
  text-align: center;
  vertical-align: middle;
}
table td img {
  text-align: center;
  vertical-align: middle;
}
#contents a {
}
</style>
<script type="text/javascript">
    hljs.initHighlightingOnLoad();
</script>
</head>
<body>
<div id="header" >
<div id="headersub">
<h1>Xiyuan Liu, Shan Huang <span style="color: #DE3737; font-size: 20px"></span></h1>
</div>
</div>


<div class="container">
<h2 style="font-family:verdana">COMP5421 / Project 4 / Multiple View Modeling</h2>
<div style="float: right; padding: 20px">
</div>
</div>


<div class="container">
<h3><b>Overview</b></h3>
<p>
	In this project we implement photometric stereo reconstruction. <br>
	The reconstructed object is observed by a fixed camera under different illumination. So we have a dense set of images to start with. The challenge is to infer a 2.5D surface description of the object (that is, a depth model), despite that the captured data are severely contaminated by shadows, highlights, transparency and that the light calibration is inaccurate. The steps of the project are:<br>
<ol>
	<li>Initial normal estimation.</li>
	<li>Normal refinement by MRF using graph cuts.</li>
	<li>Depth reconstruction by shape-from-shapelets.</li>
</ol>
</p>
</div>


<div class="container">
<h3><b>Initial Normal Estimation</b></h3>

<p><b>I. Uniform Resampling</b></p>
<p>The noisy data we have corresponds to an unevenly scattered set on the light direction sphere, which will produce undesirable biases to the result. To produce uniform sampling of light directions, we start with an icosahedron, and perform subdivision on each face 4 times recursively. It produces a total of 1313 points uniformly sampled on the light direction sphere.
<center>
	<p><img src="html/icosphere_3.png" width="40%"/></p>
</center>
<p>Suppose the object is located at the center of the light direction sphere. Ideally, we want to illuminate the object along the line joining the center and one of the vertices of the subdivided icosahedron, such that uniform distribution can be achieved. In practice, we seek the nearest light direction Lo at one vertex in the subdivided icosahedron for each captured light direction Li, and interpolate the image Io at Lo by<br>
<center>
	<p><img src="html/resampling.png" width="40%"/></p>
</center>
<p>where V is a set of indices to the captured light direction closest to Lo. Uniform resampling is thus achieved.
</p>
</p>
</p>


<p><b>II. Find Denominator Image</b></p>
<p>Complex objects contain textures and spatially varying albedos. By adopting a simple model, we choose an image that is shadow & highlight free to cancel out the surface albedo by producing <b>ratio images</b>.<br> An example is given below.<br>
<div align="center">
<table border=1>
<tr>
<td><b><font size="5">Image</font></b></td>
<td><b><font size="5">Denominator Image</font></b></td>
</tr>

<tr>
<td> <img src="html/image0001.bmp" width="100%"/> </td>
<td> <img src="html/data08_denominator.bmp" width="100%"/> </td>
</tr>
</table>
</div>
</p>


<p><b>III. Local Normal Estimation</b></p>
<p>After uniform resampling and obtaining a denominator image, we are ready to produce an initial normal for each pixel, which is needed for defining our energy functions.<br> Every pixel in an ratio image is expressed by<br>
<center>
	<p><img src="html/initial_normal_function_1.PNG" width="20%"/></p>
</center>
<p>
	 Define N = (x, y, z)T ,L1 = (l1x, l1y, l1z)T and L2 = (l2x, l2y, l2z)T. For each of the pixel in a ratio image, we obtain a plane equation: Ax + By + Cz = 0<br>
	 where A = I1l2x − I2l1x, B = I1l2y − I2l1y, C = I1l2z − I2l1z.<br>
	 Then we can solve for the initial normal (x, y, z)T by singular value decomposition (SVD) which explicitly enforces that ||N|| = 1.<br>
	 An example of initial normal is given below.
</p>

<div align="center">
<table border=1>
<tr>
<td><b><font size="5">Image</font></b></td>
<td><b><font size="5">Initial Normal</font></b></td>
</tr>

<tr>
<td> <img src="html/image0001.bmp" width="100%"/> </td>
<td> <img src="html/data08_Initial_Normal.jpg" width="100%"/> </td>
</tr>
</table>
</div>
</p>

</div>


<div class="container">
<h3><b>Normal Refinement by MRF Using Graph Cuts</b></h3>
<p>
	The energy function for graph cut minimization consists of two parts: E(N) = E_data(N) + E_smoothness(N) as specified below:<br>
</p>
<center>
	<p><img src="html/energy.png" width="45%"/></p>
</center>
<p>
	Then use gco library to construct the graph and perform energy minimization. Please refer to the paper for detailed algorithm.<br>
</p>
<div align="center">
<table border=1>
<tr>
<td><b><font size="5">Initial Normal</font></b></td>
<td><b><font size="5">Refined Normal</font></b></td>
</tr>

<tr>
<td> <img src="html/ini08.png" width="100%"/> </td>
<td> <img src="html/ref08.png" width="100%"/> </td>
</tr>
</table>
</div>

</div>


<div class="container">
<h3><b>Depth Reconstruction by Shape-from-shapelets</b></h3>
<p>Given the refined normal on each pixel, it can be transferred to gradient by the following:</p>
<center>
	<p><img src="html/stg.jpg" width="60%"/></p>
</center>
<p>After get the slant and tilt, reconstruct the surface using shapeletsurf.m from Peter Kovesi's library. The denominator image is used as texture for the model. An example is shown below.</p>
<center>
	<p><img src="html/art08.png" width="30%"/></p>
</center>
</div>


<div>
<h3><b>Sample Results</b></h3>
<div align="center">
<table border=1>
<tr>
<td><b><font size="5">Denominator Image</font></b></td>
<td><b><font size="5">Initial Normal</font></b></td>
<td><b><font size="5">Refined Normal</font></b></td>
<td><b><font size="5">3D Model</font></b></td>
<td><b><font size="5">Textured 3D Model</font></b></td>
</tr>

<tr>
<td> <img src="html/deno02.bmp" width="100%"/> </td>
<td> <img src="html/ini02.png" width="100%"/> </td>
<td> <img src="html/ref02.png" width="100%"/> </td>
<td> <img src="html/3D02.PNG" width="75%"/> </td>
<td> <img src="html/art02.png" width="40%"/> </td>
</tr>

<tr>
<td> <img src="html/deno04.bmp" width="100%"/> </td>
<td> <img src="html/ini04.png" width="100%"/> </td>
<td> <img src="html/ref04.png" width="100%"/> </td>
<td> <img src="html/3D04.PNG" width="75%"/> </td>
<td> <img src="html/art04.PNG" width="40%"/> </td>
</tr>
<tr>
<td> <img src="html/deno07.bmp" width="100%"/> </td>
<td> <img src="html/ini07.PNG" width="100%"/> </td>
<td> <img src="html/ref07.PNG" width="100%"/> </td>
<td> <img src="html/3D07.PNG" width="75%"/> </td>
<td> <img src="html/art07.PNG" width="40%"/> </td>
</tr>
<tr>
<td> <img src="html/deno09.bmp" width="100%"/> </td>
<td> <img src="html/ini09.jpg" width="100%"/> </td>
<td> <img src="html/ref09.jpg" width="100%"/> </td>
<td> <img src="html/3D09.PNG" width="75%"/> </td>
<td> <img src="html/art09.PNG" width="40%"/> </td>
</tr>
<tr>
<td> <img src="html/deno10.bmp" width="100%"/> </td>
<td> <img src="html/ini10.png" width="100%"/> </td>
<td> <img src="html/ref10.png" width="100%"/> </td>
<td> <img src="html/3D10.PNG" width="75%"/> </td>
<td> <img src="html/art10.PNG" width="40%"/> </td>
</tr>
</table>
</div>

</div>


<center><p>
	The END
<p></center>

</body>
</html>