Machine Learning Impact on Product Management

Machine Learning Impact on Product Management
Check out the Community for our current Topic at
Note: Transcripts are originally created via AI, and not all have been edited and updated.
We apologize for typos and mistakes that occur.
Full Transcript:

TOTW - Machine Learning
0:00 all right welcome back for the product growth leaders topic of the week this week we're talking about artificial intelligence and machine learning in their impacts on product management our panelists today or Greg Fenton Jason Select Steve Johnson myself and We're Gonna dig right into that too what we did the first
0:22 question so let me make myself make the sides bigger and make myself disappear
0:29 In somehow I gave myself a different room
0:32 how I did
0:35 Oh so how will machine learning impact the future of product management this was the question we posed on Friday and the first question the first answer we got was in tech it will begin to fulfill some of the number Mundane aspects of product management and Jason came in with his dissertation talking about Jessica do some those processing common components already mentioned in the short term a the challenge for softer PM to see what machine learning can be used it
1:06 i did a lot of duplicate there and used to solve problems I got our customers have also and this is where we start getting that second part also it could be used to reduce internal cost beat up interactions they bring Archimedes no solution that should be considered to solve business problems aren't his be tangible and work well in large datasets so Jason talk to me what you we talk a little bit about the Mundane aspects of product management but then you go into Solving user problems Yes I was starting to agree with the is it's interesting that Unites I kind of try to be really quick to answer this question first thing in the week and this is one of those questions actually they end up sitting on throughout the week thinking about
1:50 machine learning in so perhaps I should have waited of months I have you waited I would give you shit so off of refreshing you know i can't win this Burnside scientists try to sound intelligent amateur on a topic that's a really complex and immediately and it doesn't always work out by Earn
2:12 i don't really know gave product if machine learning can be addressed too much product management except or assessing markets assessing users assessing like usability associate with demographics and how they interact naturally machine learning as much as like using big data to find analytics as far as using machine learning actually teach a system to do something I think we just had we created from a product name is perspective perspective like what can we use big data sets to own teach teaching know what are we doing that is actually mundane It can be taught through a computer using again big data and that's key to say big data because machine learning cannot happen without large datasets and across our industries on what we do and how we do it clean very very different where the data is not actually homogenous in so that's a little tricky and saw then I started going into but ah product managers really need to think about how machine learning is evolving uM it within the industry of of what we bye to our customers in in Ali's look at that aspect into optimizing that delivery in those products in so we do have to know about machine learning but I don't necessarily think it's to speed up but were doing a public mandate was to be
3:32 so with that answer there let's go to the next destination would you get from John is not with US today on our machine learning is a tool that product managers can use to solve user problems but it does not fundamentally change a PM in a that way If you just in learning is an appropriate policy solution to your users problem that's great that means that process ave have something to do with pattern recognition categorization prediction from large amounts of data Progress may need to brush up on their math and statistics skills to get the most out machine learning if the questions about machine learning to actually do our jobs or assist us in our jobs that is different story altogether perhaps something in a pattern recognition or prediction could be helpful If we ever got datasets large enough to be useful but some somewhat proud of his judgment in nuance I met mature machine a machine could do it well steve you you respond I'm assuming here's your answer was a response to John's because you said you were once told by a product manager is that A I Am I would replace product management entirely actually let me do let me bring this up because I know you guys trying to read it you can't
4:36 I
4:38 assign Yes
4:42 I was once told by Bregman because of that I am now would replace product management highly i guess nuance and judgment are not product part of his view of product management
4:51 talk to me about where you sit where is the balance in that
4:56 how much is nuanced judgment versus how much wood data sets help us streamline and those those Mundane tasks I think the the issue is back to Jason's point is where's the Big Data
5:15 his point wise that software will become self learning you know it starts doing things like i see that you always
5:27 make your slide one fourth of the screen and then you make your image one fourth of the screen and having done that three or four or forty times I've set up an option for you to set your screen outweigh that would be great right Ah I as I look at pipe
5:50 is not a to space odyssey though
5:54 how so
5:56 The machine learning everything what that's not how you do it
6:00 well anyway my point on on big data is where's the big data Yeah I have to have the data right so you know it's like
6:11 people say Oh you should be able testing all the pages on your website near like well how do you do av testing when you have twenty five visitors a day will mean it's not like were in a google right so You're Gonna have the data to find the pattern and I do wish and there are there are some really practical data science is sorts of things that I'd love loved have like Ah If I had transcriptions of my interviews and I had say twenty interviews and transcriptions being able to run that through pattern matching and having the A I find met patterns that I did not see I'm an engineer by i've only been but
6:55 at this point I I've I find it difficult to fathom that the I could say so therefore
7:06 I'm sure it could say here's a workflow that makes more sense yep and it it can't say this workflow makes no sense
7:16 you know it's I it it tends towards
7:22 Ah common wisdom may not apply a path if I did it anyway his point was in I'm not arguing his point but his point was the machine can learn how you used Product and then update itself accordingly
7:40 which has a whole bunch of false assumptions at the beginning that is that you're not trying to use a hammer and a screwdriver yeah why it in it to me if if if Greg had the software and uses up rents I had the software each of US use it differently will update all these different stems and then word is control come from
8:02 Gregory you know it could very well be I mean like product usage for instance seems a wonderful place for Me I praying
8:12 the others I are terribly delightfully old story Wordperfect version free so only a few of US are old enough to go back there but anyway where crappy version three they in their beta program they put an a log every single non typing thing you did you know did you bold did you attack like what menu options did you choose how did you use the product for anything other than typing and to be in the beta program you had to agree to send your log when you're done
8:48 and they were surprised to find that ninety percent of the people who use wordperfect never choose quit
9:01 now and I would look to say we should just remove the quit option from the file menu
9:08 what was happening right now it is my observation so the product managers went unobserved what was happening wasn't the day on a Dos machine they would close the document they were working on reach over and just flip off the switch for the whole computer There was no shut down routine in those days Ah is likewise only ten percent of customers ever printed
9:33 and they're like should we remove print print yes well here's what was happening In those days Laser printer for five thousand dollars and only one person in the office had at your admin so you would save the file to a floppy and walk it over to your Ad NASDAQ das lots of thing would you please print this for
9:58 you so you are observational not it's observational got you're looking at is a post just pure exactly once
10:06 it is the combination of the both I I took other her persona ask back and there's a usage aspect but the raw data lacks judgment I see I took a class in business school Cobb marketing science or something like that marketing engineering and what they are positives to terms you never hear together now you don't in in what they talked about the first day of classic fights inside that classroom Why was talk about how do you use statistics in in conduit analysis and other things to make marketing decisions and the professor
10:43 quoted a study I've never been able to find this babies one of those studies that should exist but never does it says The success rate of decisions made purely on got was like thirty percent success rate of this is just be purely on analytics like forty five percent and the success on decisions made with analytics and got was like seventy five percent in it it it resonates with Me I could see that being the truth I get I don't have the full you know I could never find that that
11:13 me and sourcing Steve I could never find the source for that what are you to say greg well that's where That's where you you know the value of actually having customer experience people on with the team
11:25 because you can't rely just purely on data that's great it's a great benchmark to look at something especially run behavioral stuff but then you have to kind of stand back and go wait a minute is what is is really telling me in how my affecting like my customer experience and then and then go from there said it at your point is that you know the print function at that time you get your observation the external observation with using well call alternative data is that OH by the way It is very expensive to print
12:01 and that's you know that's that's that extra data that you need to come in and and part of that judgment of what you're doing with the product Yeah and maybe they should have actually done something instead of print maybe they should have added another function and said by the way email this to my Admin or gassing later things like that and not those are observational stuff the debt that is absolutely November Gonna get that you're never Gonna get that agreeable Yeah in in that you know then I think that you know Z You're trying to answer the why question why is somebody doing something based on the data in in you know those the that's tough to pick out from just pure data
12:48 but no now Bad you can spoil your partner is absolutely critical to have you know a large data set in order to do any type machine learning aspect which is little bit different then you know natural language processing witches You're French is an artificial intelligence that is you know we should be really clear about what which one is what wrong had a regular amplify point you made and models
13:15 Machine learning Big Data A I whatever none of them understand why and they don't nightmare so so Here's I Remember Attica data sciences class and one of the examples they use is that Walmart sells more blueberries when it's seventy degrees than they do then when it's seventy five degrees
13:38 and based on that insight from data science they say if the temperature is seventy word that store is we recommend to the store owner to move blueberries and blackberries the crown store right right we don't know why
13:55 More berries or sold at that temperature and we don't care we just know the weather forecast says is Gonna be sounded day so we need to make sure my store has allowed blackberries for blueberries which everyone I saw in it's grip and that's why some say that The addition of like that alternate data of you know because it wasn't just pure water but people buying at any given one store it was all this other data that came in that added to The solution but you're coming with an ass a product person what your trot you know you're you're at the end of the day you're trying to solve the what what are we trying to accomplish and then you're using a whole pile of tools
14:38 to help solve how I'm going to accomplish this and then you're always circling back and saying okay wait a minute did that actually give me what I wanted and then and then again and again and you know there was some statistic got so many in all machine learning projects fail
14:59 in in lot of cases it's not it's not a clarity of what we're trying to solve or you know didn't solve what I wanted to do which is okay you know it's that's fine you know that's part of in all the experimentation is that you know when we tried to use a tool we try to use all this and it really didn't solve the problem we were trying to to accomplish and that's and that's good failure is good because you learn from that as long as you're learning from failures and jump on that point the same was true twenty years ago I maybe one year ago with software yes hi how many Software projects did we began that we actually finished and actually delivered what we expected and one thing that was disturbing to me and my data science class was there was no reference made anywhere to prime minister
15:50 and for me Data science is just another form of development is just it in software development you create Greater Annie in A I or machine learning you leverage existing data love that it's program Sorting out the sharp it's Python server so that data science you're trying to come to re trying to validate hypothesis is data science is really doing that you're the hypothesis about why people are buying more blueberries in the store and now you're trying to validate why are they doing that with data science Z you know Machine learning is a very different thing the now whereas the placement of set of those blueberries best based on your observational data that comes in from cameras blah blah or the other all this other stuff is it maybe you'd have you know you can test with machine learning of different placement of things to increase performance and that's where that's where that you know What are we trying to accomplish hey okay we know this now hadaway actually increase it better right that I recognize which learning plug I challenge you there on the wide part of that we still don't hear about the why I know and if I put the blueberries next to the cash registers they don't do as well as when I put them next to the rain umbrellas and I don't sure why that is true
17:31 right so it's it it's a barrette it's a continuous AV testing you know everybody here numbers go up with it here numbers go down let's just put it here right but it's all at but it never cares why that's true it is just Embrace the is what happens it's it's ah operational data it's not particularly
17:58 well is operational let me let me get finally tipped to do a inning point of this is
18:06 I don't see the product managers have the data I don't see it that way I mean We're not informed by operational data what we're looking at is what new products should rebuild or what new persona should we saw when we can't i don't see where the data might be that we could do that pattern age wrong operational data they were seeing Mullen I that's where I think that you know the difference now is a product manager that is now working on machine learning products you really do need understand data there where the data come from the validity of your data
18:48 and also the and I wrote in in In the comments that I'll get to eventually is the ethics of the output Yeah I indeed and understand you know so you can actually get out as as you have more more data that's flown into these models and and of as a product manager understanding you don't work a good extent what the models mean you're not expected to write the models I hope hope not you wouldn't know you may know the differences between okay means you know nearest neighbor things that when it comes to some models on but you should understand how did the data show up what data you're missing and then what is the output means you can clearly state what is the output Right I figured out what are you wearing listen to your feeding state analytics you look for trends so I can make decisions as a product manager or codes consumer data when we look at machine learning it's taken a data to make repeatable actionable items to something to automate and remove a human element to it in and continuously feeding that machine and keep learning it to do better sell decision making kind of disagree process but really Automation based on the data is feeding into it so I do think that there is a large value from looking at all kinds of datasets the product managers to see trends over time users whatever the attributes are to help inform kind of decisions and go back to seize point why why is this data train this in in EM data scientist in seattle putting sources and analytics and finding those ways to provide provided data to us is invaluable However machine learning which is actually teaching machine continuously to do something I needed a product management product manager would normally do I just don't see where the problem is there that that would suck also you know or how it went or what or any consistency in the industry on how a product manager what an what a prime minister does and how he or she does it yeah so let's let's get to the final quite an end I made via greg for jumping out at the last minute but I actually jumped in the last minute to I did my into this morning miles put the deck together so as I do think that there's a big impact for solving problems in the market but machine learning is not a panacea for all problems and I want to dig into that when we get the paul about the difference between solving user problems in the providing put productivity efficiency for a product manager
21:24 but I don't want to I do think there's a near term impact that can help private managers process more information especially in a solution with lots of usage interactions and transactions to steve's point you need a lot of data to be able to do that Whether it's improving the experience you could see how people use it you could see where they go how they do it you can you could find a way for it to start as a group maybe it's not steve's usage gets optimize it's i Everybody is this where we see these transit go to these buttons that make it easier for them did it take to do what they do I identify cause it affects scenarios this is a place where it's almost a trouble shooting type thing where I've seen it work Is if you come to a common error they can start recognizing things that are happening that are leading to it maybe prevent them
22:09 or identifying this is where I was my big play as identifying new pricing and product opportunities and that I know a lot of big data firms who do a ton of work for consumer product companies talking to them about
22:25 looking through data analyzing data user data pricing data to help them optimize pricing and after my segmentation for that type of stuff
22:35 I think there's a place for it to to help us because there's only so much we can go through if you have access the data if you have access to transactions into usage information i do believe is an opportunity for to help product manager be more efficient do in process more information it was first response we got was you know the use of machine learning artificial intelligence for user experience that as they combine that was similar help and mining the data and you'll have something powerful danger is errors and you're training models but gives an opportunity for low friction Adoration particulate embedded design tools and Dev OPS
23:10 now I've worked in places where that i've consulted companies I've worked at places where we use machine learning may be more natural language processing to that point early on type stuff but clean data is also the most important thing if you can't get clean data if you can't get good libraries to train them that artificial intelligence machine learning on You're you're up a Creek I do does the wrong stuff now I'm good to go a little sideways year and will get to it a little bit up i keep coming back to
23:39 the design tools dev Ops it's about the how how much as A I am we talk about product management focus on the who the what the why and engineering designed to figure out the how
23:52 is the really answer that artificial touch machine learning is how to not a product manager tool interesting can I can I do a little rant at bell on clean they can always do a rant at city so my son and I went out to home depot
24:08 and
24:11 Butter in a wet gotta being in in winter the check out and and the lady looked me in the eye and I watched her type a five digit number
24:20 and I went wait did you just putting the zip code
24:24 and I said and she said Yeah I said but you didn't ask me what my ZiP code once and she's like Yeah marketing is making a type this step in and am Adolf I'm tired of typing gets I've just been typing my own zIP code
24:41 and I'm like well why didn't you do zero zero zero zero or none and a nun and she said oh yeah the software won't let me do that for they're Gonna open up a new store in her neighborhood right Rosa has got a a home depot and lowes and a builder Air and Anna a true value within one mile of our house but I'm still had to drive across town to get some deeper
25:05 and yet but cause the developer of the tool Yeah I Wanna ensure that we don't get around this data
25:15 they incur they they did I worst thing and that is not obviously wrong data we are now getting balls data and we can't tell until you look at it go ninety nine percent of our customers are from this ziP code an ethos and even there you might go wow we really nailed the store's location for free This is Steve this actually could be a great story for context right if you don't if you give somebody up specifications do this put these five it did then and you don't give them context for why you should do that they're Gonna put their own phone number Good point and if you put give them the contacts Hey I need you put these in because we're trying to track where our customers are coming from so we can plan where we could build another store do something else where we do our promotions all the sun with that context that changes her OH now I know I'm doing this
26:10 report without that context why AM I doing this so back to the answers Greg came in literally last minute though I'd already left the house to go pick up my cleaning so I get up A nice clean shirt on and out andalusia and I get the gum got my followed greg with an answer like oh my God let it not be one of his typical dissertations luckily it was not
26:34 an and
26:35 A respectful way or completely totally i just recall if it was a dissertation I was going to get it into the slide deck
26:43 but a short answer I could if the question is how machine learning can help prosecutors be more efficient than he agrees machine learning provides the tools to assess more information valid assumptions market the future customer needs and you go on from there
26:58 so my my other point with that spoke that's the you know the answer to the tools right out Giving US giving product managers better tools to make better judgments got it totally get it it's based on data that can come in
27:12 to help us in in multiple different ways no other point was that at where i was trying to push it towards where's the product manager fit into the entire cycle in defining what but the models are things like that that is definitely you know the machine learning specialties that come to play it you need to understand statistics are models and in how this all works is in really do
27:37 what at the end of the day where I'm coming from is that you know who is deciding you know explaining that water what's you know what what does success mean at the end of the day and can you explain how you actually came to the answer
27:55 saw when you talk about like facial recognition some things add your get in on the identifying people somebody's Gonna have to go and say this is how we came up with this problem in this is how we came up with a solution and this is you know this is where it came from
28:13 and it may not necessarily be the engineer that does then they can come up with what all the parameters that we utilized for or against all the data absolutely and we can modify parameters and things like that to come up with but you know who's actually Gonna be now at at the end of it what's Gonna happen it it's almost like you have to have acceptance criteria from product manager based on the who what why and then you can judge the output of the machine learning as who does it
28:43 me that acceptance criteria and that's where the and I think that's where the product manager role fits nicely in your working very you know this is not a you know brought over the wall then they throw it back this is a cooperative type of thing all the time and we're Gonna we're Gonna dig into that exact thing where the product manager's role this isn't our first open ended question without that's good let's get to the pole
29:07 and here make it big so people can see it and get rid of Me AH so the poll question was where will machine learning and artificial intelligence have the biggest impact on product management Steve nobody said it will replace product managers that's good i guess we agree
29:24 The the came down to two things that will be that tool to make product managers more productive that's where my vote was in or it will help solve user problems and I actually went back and forth on this surely because of I think there's a bigger impact for it to sought help solve problems but that whole vision of what Is
29:51 The solving problems of A I was really on the house side it's and that in the engineering world not in the product manager's job so I stuck with and I am once voted I couldn't change my vote so I would have gone back and forth to either but you know I think it I think that's where it's going to be mostly because I see it as a tool a how to a lotta who what why tool steve would you vote
30:15 or did you vote well I'm i did buy and I'm doing the same thing you Dare I'm not sure what I answered an easy grip describing that I was thinking about what Jason and bragged said Ah I can very much see machine learning A I don't know what we're calling us now I'm
30:35 watching user behavior and seeing friction
30:41 so as you know some of you know I'm I'M going through doing recordings of some of my course where and Ah
30:51 I can very much see that a smart program I notice that I have opened a file export it an M P three close the file opened another one and then another one and then another one and
31:10 Leaping to hey product manager I've noticed a lot of people are opening exploiting closing twenty different piles to inspire a batch export capability it was my thought were doesn't you know I used to I I I did the manual process for a while at filing there's Gotta be a better way and it was hidden away somewhere and menu but ah Yeah I've done all my videos they're all ready to go I want to drag the whole bunch into a export in say you get you just aren't you've got the whole computer I'm Gonna go Watch TV I'm Gonna come back in three hours and I want everything to be done I could see how a product manager could not spot them and machine learning curve
31:58 so you just made me think of a whole nother topic there is machine learning something that should be used for product led grow so understanding how people use it the mistakes they make and then helping create those props to help people use it better I could see that I mean my God I'm not changing my Answer again I may go back saab and mean I love and imagine Yeah Yeah Yeah I was now getting an email saying ah maybe not explicitly saying you know I notice you're doing a lot of this did you know you could do that
32:37 I can't think of an example of something came up the other day where he met gmail does a lot of that to task of denote Yeah good point to
32:47 consider that sad particularly observing what's going on you know your patterns within gmail and and making recommendations or modifications yeah what was an apple that is a machine learning or they're
33:02 using analytics to tricks
33:07 Well I got knocked out of a database
33:11 you know everybody says have a good weekend so as soon as you type have Ah and they propose you want me to finish your sentence were you good recap is that because I say it a lot or is it because everybody's at the clock
33:26 here next a next and that's where the good learning comes in play and me an auto corrects end no grammar correction things like that mom because that's that's not necessarily analytics as more a learning capability making suggestions yeah and having a good library adapted to give you best practices right Yeah here is how things are styled of what greater but the key is the feedback loop and when you don't accept the auto type or the the spell check and you keep moving forward that goes back into the large dataset and continuously teaches the programs if we do not have a feedback loop in machine learning scenario then is a one way scenario which means just analytics bright side I agree attention In it there and you know there is the there is the two aspects to that there is going back into the larger data set and then there's going into your dataset right so that's thirst you know the end personal lives annually personalized personalizing based on your patterns are what you're doing bought augmenting that with the larger pattern Beato you know that's analytics is is you know reporting what's going on machine learning actioning on out it's full circle not just I don't I a that was a great does Jason when you said that it just really hit me like wow that that's really what it is that analytics as it's sort of an output were machine learning is output into a new input that learns and changes and evolves right what was your answer did you answer the poll Yeah I answered while I have the same conundrum back and forth but I answered actually better you know for better problem solving got our problems at the end of the day
35:12 and in with exactly the you know what we're just talking about just recently is that it is you know you're you're trying to solve you know some some user needs up to a point now in I would add on that you know removing of buttons on things that that has been tried in the past with other products is year you don't use these buttons or just hide them
35:35 in just auto hide among people that that has backfired several times
35:41 in DOC different products so this there's not a certain level that you Gotta do Yeah I would also add onto the machine learning part and I think grant you talked about is the data part is that you have to be very careful about training data because you can you can tear you can build the models and you know the machine learning get stuck on on the training data not necessarily it gets and then it gets trimmed up on the real data when it actually comes in if any of you seen plenty of examples that with with the versa The type stuff and even then law enforcement using it knows I just have jason what did you vote
36:20 I think I wrote it I just find out what you voted for it and it's obvious that I feel safest way
36:28 you said that last week too I know when I called you on the fact it doesn't tell you how anybody vote until you vote there's a machine learning an interpreter a human
36:40 Yeah I think I just like them both I think Dudley scenarios in the middle there are some artifact account go back and forth any of what your what your perspective is at the time you're trying to vote on essence but I devote love the same issue in this case Yeah I'm even out at my town of my Rhetoric at the beginning of the week ever divorce I'm absorbing is around now what so what are we Gonna get into with this next
37:07 question let me go Big for everybody to read we talked about right some of the issues with machine learning and AI there are many examples of when machine learning they I go wrong and products biases and other things what Is Product management role in preventing or managing this I love the get you know Greg I would start with you on that one could we you talked a bit about a little earlier
37:35 Yeah I think that well for this is this a good one
37:40 you be because in product managers ultimate role is trying to define what are we trying to accomplish
37:48 and you know then in negotiations with The different teams is how they're Gonna actually you know execute on this the product managers role is continually saying okay what's our goal what are we trying to get to how are we doing this is it not building a rule base all set a software which which solves many many problems or is it hey let's we just nuts just shrug into these big machine learning algorithms and models armed that may very well be that I'm Gonna try and you know had a nail with a sledgehammer may not be for appropriate yeah
38:30 you think that you know sitting back and saying okay what what are we trying to do with the data that we know is is that's where product manager really sets
38:42 in a I would separate emma in all machine learning in A I A I a very different animal
38:49 trying to think more like a human that's a that's a very different animal than machine learning Yeah
38:58 steve your take
39:01 I'm sorry I went down a path with Greg bear that now I feel like I need to go into Wikipedia and get a better sense of of the different Yeah I am now are signs analytics so I feel I suddenly bill completely incompetent in and out but the current world
39:21 AH so move on okay we will move on yeah this is Jason
39:32 so I am in only we practiced there's we have a Lotta NL the Wii use internally in in I've seen presentations on your mouth interesting the Middle Google challenged and I think what we would have a problem as part of an interesting about using machine learning inside of the applications they're reading is they have to continuously evaluate the value of the machine learning hum not continuously but PETA over time is it still like mosul time we provide a feature in Emmy we know the valued possession it in then we move on a case machine learning where you have this continuous programmes teaching users are automating something What you Wanna do is make sure that are still providing value to the user community and also internally a allow machine learning is done just a kind of operationally or reduce costs in in in in my decoded are in things that nature so do that and also sampling because once that we get it a horrible situation we talk a little bit about the training data but what if you get into some some weird anomaly were all of a sudden your machine is being learned by the wrong teacher or the wrong data if you don't pay attention to that where you Gonna do is have situations in your applications that are causing are causing a outputs it you don't want or causing behaviors at you don't want in your patients so you have to be monitoring Yeah I was just actually thinking about when you were talking about that is just think about the case of like a chat bot up in out in the constant learning that the the chat bot based on the questions in in the answers are going in all it it can it can go wild now currently the learning curve again you get a setting forget it on on A I in in my answer was a squat for this was going to be sampling rate you need to as a product manager I think you're all too we build that data model of initial Reference library based on data we have every once in a while you to go back and make sure that same data is provided the same output are outcome that you're looking for because if you don't think they go off in one way or the other I think I'd got hear your environment your business environment can change drastically dumping that you can get a huge customer that's on a completely new industry that you never saw before that operates kind of very differently that's now feeding the machine what other customers in solo if your environment changes you should be looking at sampling the picture it still stays true to what your initial purpose was for this even even seasonality and I could learn your Europe a place with a high summer Prince accurate because a seasonality you turn it on in April all aside it learns stuff in the summer and it's making decisions for December which are based on the wrong things and ALEC has it has an author though so you get a lot of noise which may over time flat now but you know it could take the wrong turn are AH but the output to your point and you know it it it does go back to the sampling and then that was that was my my take is you got we have to product management has to be that the controls it's Gotta be the the book the boundaries and make sure that it still get the outcome we want it doesn't commit the learning doesn't get past the who what why and the outcome for looking for while an innovation your eerie ear canals observation that a new business comes in at that point that that's where the product manager may say well wait win this is a completely new business maybe I should actually set up a completely separate product our model for that business and not and not mixing together that that can be very valuable insight from a product point of view thinking from the customer and business yeah and a good example of that is Damn
43:22 Pa i was teaching learning the experience for us it they are I've been using them for years I'm perfectly happy I don't need your crazy security I don't have kids you know zoom bombing my sessions and stuff like that
43:39 I Wonder how much they're looking at the overall data instead of the persona based data and as a teacher for a while added there are a lot of requirements that I have that I don't him as A consultant I mean I'm doing classes of age turn people not thirty five but if I were doing very by you know I'd like it to give me the option to sort I purchased the pants by their involvement you know are they watching our day surfing the web are they playing xbox over here Annual Jason Hasn't said anything for three minutes and grant has said something a minute ago somebody call on Jason tax because he's the least engage or whatever hum an offer for him to take any move on as quickly as he did his very gay The end it but what what I anywhere it might my final thought on this whole thing in terms of product management role is a very common question I get
44:48 is should I learn to be a program but I learned a bit and I learn to code
44:54 and my answer has always been now but maybe that's because I've been a program
45:01 but I would certainly argue over the next decade that we all need to become skilled or knowledgeable in data science and all the flavors there ah I still honestly deep in my heart lee but it still programming problem
45:19 is just software creates data data science leverages data but still we need requirements me you know who what and why not the how and the other stuff right but I think they all need to be conversant in the language are data technology the way we should be in software technology
45:44 So if we look back at when we had the conversation around backgrounds for product management it's almost like we need somebody who can be a journalist and a data scientist to buy all the Dank I want aren't managers didn't Dare scientists I want them to their understand Data science is looking and data sites I get it million you need to have a an intelligent conversation but not til att not you may not necessarily need to do it but you have to understand what they're talking about like lately
46:14 Alright let's get to our second question this is
46:18 outside of the use of machine learning in A I by the fang or fame famed it depending of it's netflix or microsoft one of the most novel implementations permutations of machine learning or a you have seen in products now Jason you said you guys use some machine learning but we do really is to make the user experience a bit better just for auto filling fields We do things around coating of invoices for like geocodes and things like that and we use a previous so taking a voice type a supplier in those kind of things maybe the commodities and we used to kind of give wreck of a suggestion to the to the end user as to what was probably is Gonna be coded for so they just really have to do accepting of it but it's still have the opportunity to review it sucks not completely blind and then that feeds back into the machine then we also use it to help with approval workflows we say listen this type of you you know you ordered a computer everybody has ordered this laptop it always gets approved you know do we wanna have this automatically approve the next time since it's never not approve that kind of thing you just have like a report collapse is solid we've we've done it for religious processing components there just to make a little bit easier for the users like coding is a place where I've seen it used a time I did some work with a company that had a so eighty eight codes in aviation are like that code the doctor puts on your diagnosis code right and there's like three or four different code you could get for scientists as well same thing with eight yay that
47:56 things that have to get fixed get different codes and there's a company up in toronto that I helped to get acquired and they used they would look they would use that natural language processing on the written maintenance records too Standardized codes because if there was a heck of a chronic failure a chronic problem they needed to do that a report a a people were coding them wrong but then they took that and then pushed it back out to people saying hey This is normal coded like this and we made this change so it's interesting to see that I've seen it in the insurance industry in the health care industry as well greg at what have you seen our share I've seen it in the finance also is basically the same type as anchors the you know when you're when you're actually trading bonds and things like that across the multi currency classes you can actually bring them together using natural language processing things like that but back into the thumb seen so many examples but a avid pick one that in i'll go back to the trading systems is that there's a there's an increased usage of machine learning around all the data that's making up any given type of OM Nom security and helping traders may those instantaneous decisions on what to trade and what about what the balances are in the risks associated with that hum and using lots of data from trends past analysis of of past trades pass data in bringing those to a forefront very very quickly
49:30 to make that to help and it's not actually making the trade but it actually puts it up in front of the traitor to actually say here's what we seen you could decide what to do with it
49:42 and it's just got a constant they are you know a loop that's going on based on what information goes on and it's very fast operation to do that and I've I've seen stuff at a very slow operation OH I was working with the company up in Montreal who does machine learning AI for underground Hardrock mines and they they sniff all the data on A Piece of machinery so they understand all the talent tell metric type stuff on the big equipment in the drills in that type of stuff and they're able to his work backwards from when a break happens a fault happened something goes wrong and read the data and start getting it's ass or the all predictive we could tell you you're gee had a commercial they knocked on the door to fix refrigerator how you don't need to be fixed oh it told us it needed dina this is the type of stuff they were doing there is pretty cool because they could say who this asset In the mine is Gonna break soon let's pull it out now so we don't lose whatever time or let's do this type of maintenance while the truck is out because we know what's about to to do that type of stuff is pretty cool steve what what have you seen
50:50 now I don't know anymore what I'm saying you know I mean
50:55 gmail comes up and says you sent an email to grab the other day I haven't heard back you want you know here's a reminder that you might want to follow up with that Yeah Ah it's it's it's creeping into my life in ways that I don't notice a except I just got out then asking indians and a buddy of mine is building a website right now with wicks and he did it three or four pages that were basically copies of one another and which came up and said if you're Gonna keep doing this you need to learn dynamic pages
51:32 because you're basically
51:35 If if if you got all these pages that saying Iranian dynamic pages is is is it is a table driven page and is necessary information from stretchy Ah and I'm like
51:46 that's Cool I already having the software suggests that there's something new a new shirt learn
51:55 product they weren't making it easier to do my job now what I see is that UM in multiple things are Gonna make a really exciting over time so The number of devices that are becoming more and more edge intelligent plus the combination of having access to large stores and a network that is high speed a reliable over like five gee you're Gonna see a tremendous amount of data that comes in and you'll be able to fire those back into you know it the the actual engines themselves of actually be split so you'll have your client side machine learning but also augmented with server side machine learning of crunching of data and you're Gonna see these things rapidly go very quickly I would see that happening quickly on on cars as well as like cameras and and home security systems things like that will happen very very quickly
52:59 you know I think five G's just gonna change just because to access the data is Gonna be like it's just you're just Gonna have data everywhere Yeah on but the downside of that is you're Gonna have data everywhere and you're Gonna have to mute know who's going to decide this is garbage data just because act on it that's a great line addition to the rapid fire question we have here
53:25 throwing lightning round question what are the product management tasks that are most Most likely to gain value from machine learning in A I and it seems like Greg you're just talking there about the more data you get it's great because you got the data but it's terrible because you got all his data how do you do it I'd love to get you start with you on this question I think the reno just to add onto what I'm saying is that you know because you have all this information now what of the different solutions you're Gonna have for for customers' needs you know we you you talked about in oH predictive predictive failure rates long for mining equipment that that occurs on airlines and and everything everywhere it's really expensive you're Gonna see that many many cases where the two cars Gonna get smarter smarter smarter for you or home security or you know everything around kind of like if you put a pie chart around your life is like the you know you're Gonna start seeing machine learning helping you with decisions around your entire life
54:25 that that may be helpful or may not be but that's where a product managers are going to come in and say now that I have all this what what are the solutions that I want for customers and it's just Gonna become bigger and bigger
54:39 and who apply we didn't really talk about but greg you were implying strongly Is There's also pretty quick lay you know a privacy ethical scenario here that we're all ready and I know I I I feel like googles good
54:58 and you know first do no harm or whatever their their slogan is so they've got a lot of my data and i've got a nasty not got you know they're they're home automation stuff but I'M messing around with other kinds of home automation through my Amazon devices and it's starting to freak me out I'm Gonna get this third party app that's written it has like Chinese him in Mandarin or something in the copyright notice and they're like what is your email what is your wifi Password and I'm like who AM I telling this to
55:33 so Yeah I've returned almost everything I've bought from Amazon that wasn't made by Amazon and I'm pretty sure I Trust analysed I don't trust Amazon or Google I I I am using apple as my name Data source outside of GMAIL for work and impart personal but I know too many times we
55:55 expect with amazon when we had our whatever plugged in it was like getting a little creepy
56:01 Steve Slaton I think that's where it comes down to there from a product point of view where's where's that line that that's helpful versus creepy in that that's that's the interesting part in it you know I think that you're actually writes Davis ethics and privacy become become really interesting that's why I say that thing or things Gonna get thrown to the edge where you keep within your own private system
56:25 Java enough they could guarantee you that all your data was actually only within your device and never got out and then that's it that's a different thing which is where apple comes from down verses like hey we're just Gonna take all is anonymous information and you know use our big big machines to help augment that right now That's a bigger and better for everybody Yeah but you know but you know I think the privacy really becomes fascinating it's a fascinating it is bad area okay lightning round Sarah I'm saying Yeah October the product management tasks most likely became value from this would be story mapping I mean just understanding the journey with in the product and then from a marketing standpoint the same pit fire journey mapping
57:20 locate Jason now out have to agree with steve since I'm really having a hard time
57:26 game where we can got brought members except for obviously know giving more in our toolkit to think about aware in with the applications we create where this to be utilized and mcGregor say
57:39 and make sure that we provide a value to do that not just the crate him Alpha male sick then greg any other an era thought an answer that's actually correct characters like you know it's it's it's a tool so hey will just create another machine learning I completely agree Jason's were just why not in a sling should if purpose it has to be part but purpose of our up I'm going to say work is Gonna help the most is in trouble troubleshooting and usability Because it's Gonna be able to see the trends of where people get stuck and how they get stuck and why they get stuck in weather is putting it back into a product that growth thing of giving people suggestions maybe goes into the story the journey of how they use the software but I think that it's Gonna help a ton on on that front helping to make it more usable because you're you're each time you see transients and data happening as can help improve how I it's clipping right oh you're trying to
58:33 you're stuck here let me tell you this right then adding each time because it can go for it further with that I you know what with that said were at the end of our conversation we do this every week quests day In the computer all on Wednesday and this incredibly great conversation on the Friday I appreciate you Guys your time your insight and Input
59:02 Always a great time always my favorite discussion that actually this is one where you know trustees point Yeah gregg at we're probably Gonna do a a webinar on data and machine learning and AI and help us understand better everything that they're but I always pushes me to think and learn and I change my opinions because I'm listening to people who I value their their thought so thank you guys so much have a great weekend and will second overtime make like a snake you Bye