Machine Learning and the Camera Sensor

One of the larger themes I’ve been integrating in industry conversations with tech leaders and executives is how giving computers the ability to see and hear us will define a new era of computing. All the advancements in machine learning the past few years have set a new foundation to give computers the ability to learn on their own. A computers ability to learn depends entirely on the information available to it. This data has largely been textual but we are on the cusp of the computing being able to utilize all its sensors to learn as well. This includes the microphone, camera sensor, GPS, accelerometer, and any other host of optional sensors available to integrate into our computers. I do believe, however, the camera sensor will be the one which will lead to the most obvious customer value of machine learning and AI.

The camera sensor will be one of those unique areas for machine learning in consumer applications because it will only see what consumers allow it to see. In particular around the smartphone camera which is not utilized until consumers use an application which activates the sensor. Given the smartphone is the most personal computer people own, it is the camera sensor in the smartphone which could yield the most consumer value when it comes to machine learning and AI which benefits its owner. With that in mind, it is interesting to understand what consumers use their camera sensor for most often.

We at Creative Strategies, recently did a US-based study among consumers to understand the role the smartphone plays in photography. When we scoped out this study our goal was broadly looking at smartphone photography and I hadn’t anticipated a particular finding that got me thinking more about this machine learning theme. Below is a graph from our study showing the things US consumers take pictures of most often.

In general, you will find the position of most of these results fascinating. Particularly, how low selfie’s rank overall on the list. Selfies are the one use case where we observe publicly people do most often. So if you are like me you assumed selfies would rank more toward the top than the bottom of this list. Granted, what I’m showing you are the top-line results from our study. If I was to break this out by demographics of both age and gender it would like a bit different. For example, taking selfies of both the individual and with someone else, ranks as the third and fourth on this list for those 18-25 yrs old.

The thing that caught my up the most was the second most photographed thing on the list which is “information I need to remember.” If you examine your own usage you probably reaize you take pictures of lists, or things at a store you want to remember, or of homework or meeting notes on a whiteboard, etc. We may not realize just how often we do this because it isn’t the most exciting thing to photograph but it highlights how often we use our smartphone camera as a way to augment our memory.

This, to me, seems one of the more obvious opportunities for machine learning. What if our smartphone could recognize that what we have photographed is designed to help us remember something and take action on our behalf to help make that happen. Maybe if we take a picture of a homework list or meeting notes on a whiteboard the camera will take that information and automatically create the contextual notes or even itemize a to-do list for us when relevant. The clear challenge here for machine learning will be to truly understand the context of why we took the photo and what we want to remember. This could be where a host of data comes into play from my location, what is on my calendar like the context of the meeting I’m in, for example.

The other challenge that stands in front of machine learning in this use case is the personalized nature of this particular use case I’m describing. Machine learning does best with large communal data sets. This is why teaching a computer to recognize a dog or a tree is easy since it can take all images of dogs and trees to train the computer. The bigger challenge here is to teach a computer what is my dog vs any dog. This is where the localized, on-device, nature of these algorithyms will come into play and an area where I think Apple’s approach will yield fruit. Apple seems much more interested in training your iOS/OS X computer to learn about its owner specifcally than it is about more general machine learning like Google is doing. Apple is actually undergoing the more difficult of machine learning tasks of the two companies. Making machine learning personal is new, but also exceptionally difficult.

While this is one area to watch with regard to machine learning and the camera sensor, the other will be ways that the computer can make your photos better. This is single handedly the most interesting thing Google is doing with the Pixel 2/XL and their camera technology. I’ll be doing a deeper dive on the Google Pixel 2 camera in the coming weeks, but from my few weeks taking a range of photos with it, seeing how Google’s machine learning around imaging comes into play to eliminate noise in low-light, balance colors, apply post processing of blur or shades, and more is very interesting.

I point this out because in our smartphone study we asked consumers some of the things they are interested in as features for the camera or camera software and ranking near the top of the list was having smarter technology to help them take better pictures. Adopting machine learning into the photography process will only help people take better pictures and add value in information gathering and collecting. As I said earlier, machine learning around the image sensor is going to be a key battle ground for both operating systems and apps as well and will be the easiest place to see machine learning manifest itself for consumer benefit.

Published by

Ben Bajarin

Ben Bajarin is a Principal Analyst and the head of primary research at Creative Strategies, Inc - An industry analysis, market intelligence and research firm located in Silicon Valley. His primary focus is consumer technology and market trend research and he is responsible for studying over 30 countries. Full Bio

82 thoughts on “Machine Learning and the Camera Sensor”

  1. Hi there! I could have sworn I’ve been to this blog before but after looking at a few of the posts I
    realized it’s new to me. Nonetheless, I’m certainly delighted I
    stumbled upon it and I’ll be bookmarking it and checking back regularly!

  2. Hey there, You’ve done an incredible job. I’ll certainly digg
    it and personally suggest to my friends. I’m sure they will be benefited from this site.

  3. You are so cool! I do not think I have read anything like this before.
    So nice to discover somebody with some genuine thoughts on this subject.

    Really.. many thanks for starting this up. This site is something that’s needed
    on the internet, someone with some originality!

  4. Hey I know this is off topic but I was wondering if you knew of
    any widgets I could add to my blog that automatically tweet my newest twitter updates.

    I’ve been looking for a plug-in like this for quite some
    time and was hoping maybe you would have some experience
    with something like this. Please let me know if you run into anything.
    I truly enjoy reading your blog and I look forward to your new updates.

  5. Hmm it looks like your site ate my first comment (it was super long) so I guess I’ll just sum it up what I submitted and say, I’m
    thoroughly enjoying your blog. I as well am an aspiring blog blogger but I’m still new to everything.

    Do you have any suggestions for newbie blog writers?
    I’d really appreciate it.

  6. Good way of telling, and fastidious piece of
    writing to take data about my presentation focus, which i am
    going to convey in institution of higher education.

  7. I’m really enjoying the design and layout of your site.
    It’s a very easy on the eyes which makes it much more pleasant for me to come here and
    visit more often. Did you hire out a designer to create your theme?
    Superb work!

  8. I do not even know the way I ended up right here,
    however I thought this publish was great. I do not
    recognize who you’re however definitely you’re going to a famous blogger in case you aren’t
    already. Cheers!

  9. Hello there, You’ve done a great job. I will certainly digg it and personally suggest to my friends.
    I’m sure they will be benefited from this web site.

  10. Hi there, for all time i used to check weblog posts here in the early
    hours in the break of day, as i like to learn more and more.

  11. Hi there! This is kind of off topic but I need some guidance from an established blog.
    Is it very hard to set up your own blog? I’m not very techincal but I can figure things out pretty fast.
    I’m thinking about setting up my own but
    I’m not sure where to begin. Do you have any tips or suggestions?
    With thanks

  12. Hello there! This is kind of off topic but I need some guidance from an established blog.
    Is it hard to set up your own blog? I’m not very techincal but I can figure things out pretty quick.
    I’m thinking about creating my own but I’m not sure where to start.

    Do you have any ideas or suggestions? Many thanks

  13. We’re a gaggle of volunteers and starting a brand new scheme in our community.
    Your site provided us with useful information to work
    on. You’ve performed an impressive process and our whole community
    will probably be thankful to you.

  14. If you are going for most excellent contents like I do, simply pay a quick visit this
    site all the time for the reason that it presents feature contents, thanks

  15. What i do not understood is actually how you’re
    now not really a lot more well-appreciated than you may be right now.
    You are so intelligent. You realize therefore significantly with regards to this
    topic, made me individually imagine it from a lot of various angles.
    Its like women and men aren’t involved except it’s something
    to do with Woman gaga! Your personal stuffs nice. All the time take care of it up!

  16. I just could not go away your site prior to suggesting that
    I actually loved the usual info an individual supply on your guests?
    Is going to be back frequently to check up on new posts

  17. If some one needs to be updated with newest
    technologies afterward he must be pay a quick visit
    this web site and be up to date everyday.

  18. Hey there! This is kind of off topic but I need some advice
    from an established blog. Is it hard to set up your own blog?
    I’m not very techincal but I can figure things out pretty quick.
    I’m thinking about making my own but I’m not sure where to begin. Do you have any ideas or suggestions?
    Cheers

  19. I am truly happy to glance at this web site posts which contains tons of
    valuable information, thanks for providing such data.

  20. Great post. I used to be checking constantly this weblog and I am impressed!
    Extremely useful information particularly the ultimate phase :
    ) I take care of such info much. I used to be seeking this particular info for a very long
    time. Thanks and good luck.

  21. Have you ever thought about writing an ebook or guest authoring
    on other blogs? I have a blog based upon on the same ideas you
    discuss and would love to have you share some stories/information. I know my audience would value
    your work. If you’re even remotely interested, feel free to send
    me an e mail.

  22. I like the valuable information you provide to your articles.

    I will bookmark your weblog and test again right here regularly.
    I am quite certain I’ll be informed many new stuff proper right here!
    Good luck for the next!

  23. Hello! I just wanted to ask if you ever have any problems with hackers?
    My last blog (wordpress) was hacked and I ended up
    losing a few months of hard work due to no data backup. Do you have any solutions to stop hackers?

  24. What’s Going down i’m new to this, I stumbled upon this I have found
    It positively useful and it has aided me out loads. I hope to contribute & aid other customers
    like its aided me. Good job.

  25. Hi, i think that i saw you visited my weblog
    thus i came to “return the favor”.I am attempting to
    find things to improve my web site!I suppose its ok to use a few of your ideas!!

  26. Asking questions are truly good thing if you are not understanding something
    fully, except this post presents good understanding even.

  27. I have read so many articles regarding the blogger lovers but this
    piece of writing is truly a fastidious piece of writing, keep
    it up.

  28. Hi, I do think this is an excellent blog. I stumbledupon it 😉 I am going to revisit once again since i have book-marked it.
    Money and freedom is the best way to change, may you be rich and continue
    to guide other people.

  29. I ɑm eҳtremely impressed with your writing skills and alz᧐ with the lаyout on your blog.
    Is this a pɑіd theme oor diԀ you modify it yourself?
    Anyway keep up the nice quɑlity writing, it’s
    rare to see a nice blog like this one today.

  30. Niϲe blog! Is your theme custom made or did yoᥙ download it
    from somewhere? A design like yours with a few simple tweeks would really make
    my bloog staand оut. Please let me now where үou got your theme.
    Thanks

  31. I like the valuable information you provide in your articles.
    I will bookmark your blog and check again here frequently.
    I’m quite certain I’ll learn a lot of new stuff right here!
    Good luck for the next!

  32. Cоool blog! Is youг theme custom made or Ԁid you download it from somewhere?

    A design like yours with a few ѕimmple aԀjustements woսlԀ really make my blog stand out.
    Pleaѕe let me know where you got ʏour design.
    Kudos

  33. When someone writes an article he/she keeps the image of a user in his/her brain that how a user can know it.

    So that’s why this post is outstdanding. Thanks!

  34. Woᴡ, wondsrful blog structure! How lengthy һave
    you beden running a blog for? you made runnіng a blog look easy.
    The whole look of your ѕie is magnificent,
    let alone the content materіal!

  35. What’s up it’s me, I am also visiting this web page on a regular basis, this site is
    actually fastidious and the people are in fact sharing fastidious
    thoughts.

  36. Hey there fantastic blog! Does running a blog like this take a large
    amount of work? I have no understanding of programming however I had been hoping
    to start my own blog in the near future. Anyhow, if you have
    any recommendations or tips for new blog owners please share.
    I know this is off subject however I simply needed to ask.
    Thank you!

  37. With havin sо muych written content do you ever rᥙn into аnny
    problems of plagorism ⲟr copyright infringement?
    My blog һas a lot of unique content I’ve either authored myself
    or outsourced but it appears а lot of it iis popping it
    up all over thhe web without my permіssion. Do you know ɑnyy ways to
    help protect ɑցainst content from being ripped off?

    I’ɗ really appreciate it.

  38. Thᴡnks on your marvelous posting! I quite enjoyed reading it, уou could be a grеat author.

    I wilⅼ make cеrtain to bookmarkk your blog and will eѵentually come back later in life.
    I want to encourage that you continue your gгeat job, ave a nice weekend!

  39. When I originallу commjented І clicked the “Notify me when new comments are added”
    chеcқbox and now each time a cߋmment iѕ added I
    get severɑl emailѕ with the same cоmment.
    Is here any waay you can remove people from thаt service?
    Appreciate it!

  40. Great post however I was wanting to know if you could write a
    litte more on this subject? I’d be very grateful if you could elaborate a little bit more.
    Cheers!

  41. Ꮋey there! Ӏ know this іs kinda off topic however I’ⅾ figured I’d ask.
    Would you be interеsted in exchanging links oor mаybe guest writing a blog post or vice-versa?
    My blog addresses a lot of thhe same ѕubjects as yours and I think wwe
    ϲould greatly benefit from each other. If you happen to be interested feel free to shoot
    me an email. I looқ forward tߋ hearing from you!
    Terrific Ƅlog by the wɑy! https://online-learning-initiative.org/wiki/index.php/User:ZUXFiona2178

  42. Hey there! I knoԝ this is kindda off topic however
    I’d figured I’d ask. WoulԀ you bee іnterested iin exchanging links or maybe guest wгiting a blog post
    or vіce-veгsa? Ꮇy blog addresses a lot of
    the same subjects as yοurѕ and I think we could greatly
    benefit from each othеr. If yoou һаppen to Ƅe
    interested feel free to shoot me an emaiⅼ. I look forward to
    hearing from you! Terrifkc blog Ƅy the way! https://online-learning-initiative.org/wiki/index.php/User:ZUXFiona2178

  43. hey there and thank you foor youг informɑtion – I’ve definitely picked upр something nnew rom right
    here. I dіd however expertise some technical issues usіnmg thіs site, since I experienced to reloaԁ
    the web site lots of timkes previous to I coul get it to load properly.
    I haⅾ been wondering iff your hostinng is OK? Nоt that I’m complaining, but slow loading instanceѕ times wiⅼl sometimeѕ affect your placement in gⲟogle and cɑn damage your quality
    score if ads and marketing with Aɗwords. Well I’m addіng this RSЅtoo
    my emwiⅼ andd could look out for a lot more of your respective fascinatіng content.
    Ensure tһaat you update this again soon. https://bbarlock.com/index.php/User:HassieThalberg0

  44. hey tһere and thank you forr yoᥙr iinformаtіon – I’ve definitely pickеd uρ something
    new from right һere. I did hߋwever expertise somе technical issues using this site,
    since I experienced to reload the webb site lots ߋf times рrevious to I coulⅾ geet it to loаɗ pгoperly.

    I had been wondering if your hosting is OK? Not that
    I’m complaining, but slow loading istances times will sometіmes affect your
    placеmsnt in google and can damage үour quality score if
    ads аnd marketing with Adwords. Well I’m addіng this RSS to my email and could
    looқ out for a lot more of your respective fascіnating content.
    Ensᥙre tһat yyⲟu update this again soon. https://bbarlock.com/index.php/User:HassieThalberg0

  45. Hi, i reaԁ your blog from time to time and i own a similаr one and i was juѕt wonderіng if
    you get a lot of spam comments? If so how do you reduce it, any pⅼkugin ߋrr anything yoᥙ can suggest?
    I get so mucһ lately it’s driving me insane so anny sipport is νery much appreciated.

  46. Pretty nice post. I just stumbled upon your weblog and wanted to say
    that I’ve truly enjoyed surfing around your blog posts.

    In any case I’ll be subscribing to your feed
    and I hope you write again soon!

  47. I am curious to find out what blog system you’re using?
    I’m experiencing some minor security problems with my latest site and I’d like to find something more safe.
    Do you have any recommendations?

Leave a Reply

Your email address will not be published. Required fields are marked *