Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Crazy computer vision stuff in the browser
Zeno Rocha
April 28, 2015
Programming
5
850
Crazy computer vision stuff in the browser
Zeno Rocha
April 28, 2015
Tweet
Share
More Decks by Zeno Rocha
See All by Zeno Rocha
The Next Generation of Developer-First Products
zenorocha
1
450
7 Habits of Highly Productive Developers
zenorocha
1
300
7 Hábitos de Desenvolvedores Altamente Produtivos
zenorocha
1
340
What's new in the Liferay Community
zenorocha
0
580
Launching Liferay Projects Faster with WeDeploy
zenorocha
1
470
How Liferay fits into the real of latest technologies
zenorocha
0
440
Estoicismo e JavaScript
zenorocha
3
920
Por que ninguém se importa com seu novo projeto open source?
zenorocha
2
800
Como investir em... você!
zenorocha
1
490
Other Decks in Programming
See All in Programming
Android開発者のための Kotlin Multiplatform入門
ntaro
0
190
開発トップのマネジメント: 採用からオンボーディングで始まる一貫した生産性向上戦略
haruki_maejima
PRO
3
730
Android Jetpack supports KMP
fornewid
0
120
CSC307 Lecture 07
javiergs
PRO
0
200
Folding Cheat Sheet #7
philipschwarz
PRO
0
150
みんなのオブザーバビリティプラットフォームを作ってるんだがパフォーマンスがやばい #mackerelio #srenext
ne_sachirou
0
350
APIのない大学ログインWebサービスをWKWebViewとJavaScriptでアプリ化した話
akidon0000
1
330
英語
s_shimotori
1
220
Composing an API the *right* way (Droidcon Berlin 2024)
zsmb
1
440
社内 LT 会を発足し、アウトプット文化を醸成させるために考えたこと・やったこと / Starting internal LT meetings and fostering an output culture
mackey0225
3
110
feature環境をGitHub ActionsとCloudFormationでいい感じに管理する
nealle
2
310
最近追加した型の紹介とその振り返り
aki19035vc
0
150
Featured
See All Featured
Cheating the UX When There Is Nothing More to Optimize - PixelPioneers
stephaniewalter
277
13k
Build your cross-platform service in a week with App Engine
jlugia
227
17k
Making the Leap to Tech Lead
cromwellryan
127
8.7k
Statistics for Hackers
jakevdp
792
220k
Refactoring Trust on Your Teams (GOTO; Chicago 2020)
rmw
29
2.5k
[Rails World 2023 - Day 1 Closing Keynote] - The Magic of Rails
eileencodes
17
1.5k
What's in a price? How to price your products and services
michaelherold
239
11k
Fight the Zombie Pattern Library - RWD Summit 2016
marcelosomers
228
16k
Scaling GitHub
holman
458
140k
jQuery: Nuts, Bolts and Bling
dougneiner
61
7.3k
Producing Creativity
orderedlist
PRO
340
39k
Code Reviewing Like a Champion
maltzj
517
39k
Transcript
CRAZY COMPUTER vision stuff in the browser
.html .css .js
zenorocha
None
None
None
None
None
computer vision ? what is
Persistence of vision
Persistence of vision
Motion perception
Motion perception
vimeo.com/108331968
Face detection
Face recognition
Autonomous robots
Autonomous robots
Virtual reality
Augmented reality
Augmented reality
how to do that?
None
None
how to do that in the browser?
1. Access the camera
None
Request user’s webcam navigator.getUserMedia({ video: true }, onSuccess, onFail);
Request user’s webcam navigator.getUserMedia({ video: true }, onSuccess, onFail);
Request user’s webcam navigator.getUserMedia({ video: true }, onSuccess, onFail);
2. Reproduce in a video
<video>
Get User Media callback function onSuccess(stream) { var video =
document.querySelector('video'); video.src = URL.createObjectURL(stream); video.onloadedmetadata = function(e) { // Done }; }
Get User Media callback function onSuccess(stream) { var video =
document.querySelector('video'); video.src = URL.createObjectURL(stream); video.onloadedmetadata = function(e) { // Done }; }
Get User Media callback function onSuccess(stream) { var video =
document.querySelector('video'); video.src = URL.createObjectURL(stream); video.onloadedmetadata = function(e) { // Done }; }
Get User Media callback function onSuccess(stream) { var video =
document.querySelector('video'); video.src = URL.createObjectURL(stream); video.onloadedmetadata = function(e) { // Done }; }
3. Obtain pixel matrix
<canvas>
4. Tracking algorithms
Request accepted Request access to the camera Reproduce camera <video>
Obtain pixel matrix <canvas> Tracking algorithms Result Summary
tracking.js
None
Maira Bello Team eduardo lundgren java pablo carvalho zeno rocha
100% JavaScript! Open source No dependencies Simple and intuitive API
Automatic setup Built-in tracking algorithms Easily extensible High performance Unit & performance regression tests ~ 7 Kb
tracker color
Demo
var tracker = new tracking.ColorTracker('magenta'); tracking.track('#video', tracker, { camera: true
}); tracker.on('track', function(event) { event.data.forEach(function(rect) { // Your code goes here }); });
var tracker = new tracking.ColorTracker('magenta'); tracking.track('#video', tracker, { camera: true
}); tracker.on('track', function(event) { event.data.forEach(function(rect) { // Your code goes here }); });
var tracker = new tracking.ColorTracker('magenta'); tracking.track('#video', tracker, { camera: true
}); tracker.on('track', function(event) { event.data.forEach(function(rect) { // Your code goes here }); });
tracker object
Demo
var tracker = new tracking.ObjectTracker('face'); tracking.track('#video', tracker, { camera: true
}); tracker.on('track', function(event) { event.data.forEach(function(rect) { // Your code goes here }); });
var tracker = new tracking.ObjectTracker('face'); tracking.track('#video', tracker, { camera: true
}); tracker.on('track', function(event) { event.data.forEach(function(rect) { // Your code goes here }); });
var tracker = new tracking.ObjectTracker('face'); tracking.track('#video', tracker, { camera: true
}); tracker.on('track', function(event) { event.data.forEach(function(rect) { // Your code goes here }); });
tracker custom
var MyTracker = function() { MyTracker.base(this, 'constructor'); }; tracking.inherits(MyTracker, tracking.Tracker);
MyTracker.prototype.track = function(pxs, width, height) { // Your code here this.emit('track', { // Your code here }); };
var MyTracker = function() { MyTracker.base(this, 'constructor'); }; tracking.inherits(MyTracker, tracking.Tracker);
MyTracker.prototype.track = function(pxs, width, height) { // Your code here this.emit('track', { // Your code here }); };
var MyTracker = function() { MyTracker.base(this, 'constructor'); }; tracking.inherits(MyTracker, tracking.Tracker);
MyTracker.prototype.track = function(pxs, width, height) { // Your code here this.emit('track', { // Your code here }); };
var MyTracker = function() { MyTracker.base(this, 'constructor'); }; tracking.inherits(MyTracker, tracking.Tracker);
MyTracker.prototype.track = function(pxs, width, height) { // Your code here this.emit('track', { // Your code here }); };
var tracker = new tracking.MyTracker(); tracking.track('#video', tracker, { camera: true
}); tracker.on('track', function(event) { event.data.forEach(function(rect) { // Your code goes here }); });
var tracker = new tracking.MyTracker(); tracking.track('#video', tracker, { camera: true
}); tracker.on('track', function(event) { event.data.forEach(function(rect) { // Your code goes here }); });
var tracker = new tracking.MyTracker(); tracking.track('#video', tracker, { camera: true
}); tracker.on('track', function(event) { event.data.forEach(function(rect) { // Your code goes here }); });
who is using it?
kdzwinel.github.io/JS-face-tracking-demo
thedevilini.com/#masks
lessons learned
24 fps 60 fps 1000ms / 24fps = 41ms per
frame
24 fps 60 fps 1000ms / 24fps = 41ms per
frame
None
None
None
Int32Array Uint16Array Float64Array Uint8ClampedArray [ ] Typed arrays
jsperf.com/tracking-js-arrays
jsperf.com/tracking-js-arithmetic
Demo
None
None
web components > > -
how we create a component nowadays?
1. Never create! Just use a plugin
2. Copy & paste someone’s code
3. And hope it works
3. And hope it works
web components
web components Templates
web components Templates Custom Elements
web components Templates Custom Elements Shadow DOM
web components HTML Imports Templates Custom Elements Shadow DOM
None
var tracker = new tracking.ColorTracker('magenta'); tracking.track('#video', tracker, { camera: true
}); tracker.on('track', function(event) { event.data.forEach(function(rect) { // Your code goes here }); }); Remember?
Using Web Components… <video is="video-color-tracking" target="magenta" camera="true" ontrack="onResult" </video>
more ?
trackingjs.com
Where the magic happens Your comfort zone
Thanks! zenorocha.com @zenorocha
Thanks! zenorocha.com @zenorocha