Weekly QuEST Discussion Topics 22 July

QuEST 22 July 2016

This week we will have a discussion from a colleague who is in the area working with our cyber guys – but his company is focused on the issues in making natural language processing useful in items we deal with daily (example cars, appliances, …).  As we’ve been discussing Question/ Answer systems we will use this target of opportunity to talk with someone trying to transition this technology.

Mycroft.

Mycroft is the open source community’s answer to Siri, Cortana, Google Now and Amazon Echo that is being adopted by the Ubuntu Linux community.  The technology allows developers to include natural language processing in anything from a refrigerator to an automobile.  We are developing the entire stack including speech to text, intent parsing, skills framework and text to speech.  The team is beginning to make extensive use of machine learning to both process speech and determine user intent.  We have a very active user community and are working with students at several universities to improve and extend the technology.  They got started by pitching a product through Kickstarter and now have deals to be included in the base install of upcoming Ubuntu distributions.  It will be interesting to see how the open source community develops and forks their codebase compared to how the Google’s and Apple’s develop theirs.

Home Page: https://mycroft.ai/

Kickstarter: https://www.kickstarter.com/projects/aiforeveryone/mycroft-an-open-source-artificial-intelligence-for

Kickstarter YouTube: http://ostatic.com/blog/mycroft-a-startup-is-focusing-on-open-source-ai-for-the-home

News: http://ostatic.com/blog/mycroft-a-startup-is-focusing-on-open-source-ai-for-the-home

News: http://news.softpedia.com/news/mycroft-uses-ubuntu-and-snaps-to-deliver-a-free-intelligent-personal-assistant-506097.shtml

News: http://linux.softpedia.com/blog/mycroft-ai-intelligent-personal-assistant-gets-major-update-for-gnome-desktops-506207.shtml

news summary (20)

Categories: Uncategorized

Weekly QuEST Discussion Topics and news, 15 July

After the recent deadly Tesla crash while on autopilot – and related articles several questions arise – we want to have a discussion on these topics:

When is AI appropriate?
What is the technical debt in a machine learning approach?
Concrete Problems in AI safety?

https://www.technologyreview.com/s/601849/teslas-dubious-claims-about-autopilots-safety-record/?set=601855

Tesla’s Dubious Claims About Autopilot’s Safety Record

Figures from Elon Musk and Tesla Motors probably overstate the safety record of the company’s self-driving Autopilot feature compared to humans.

Tesla Motors’s statement last week disclosing the first fatal crash involving its Autopilot automated driving feature opened not with condolences but with statistics.

Autopilot’s first fatality came after the system had driven people over 130 million miles, the company said, more than the 94 million miles on average between fatalities on U.S. roads as a whole.

Soon after, Tesla’s CEO and cofounder Elon Musk threw out more figures intended to prove Autopilot’s worth in a tetchy e-mail to Fortune (first disclosed yesterday). “If anyone bothered to do the math (obviously, you did not) they would realize that of the over 1M auto deaths per year worldwide, approximately half a million people would have been saved if the Tesla autopilot was universally available,” he wrote.

,,,

When AI? … The short version of my answer is, AI can be made appropriate if it’s thoughtfully done, but most AI shops are not set up to be at all thoughtful about how it’s done. So maybe, at the end of the day, AI really is inappropriate, at least for now, and until we figure out how to involve more people and have a more principled discussion about what it is we’re really measuring with AI

What is technical debt and how does this idea apply to the AI problem?  The explanation I gave to my boss, and this was financial software, was a financial analogy I called “the debt metaphor”. And that said that if we failed to make our program align with what we then understood to be the proper way to think about our financial objects, then we were gonna continually stumble over that disagreement and that would slow us down which was like paying interest on a loan.

That leads to a discussion on issues when machine learning makes mistakes – In this paper we discuss one such potential impact: the problem of accidents in machine learning systems, defined asunintended and harmful behavior that may emerge from poor design of real-world AI systems.  – from the paper :  Concrete Problems in AI Safety – from google brain / Stanford / UC Berkeley – Amodei et al

 

First, the designer may have specified the wrong formal objective function

  • such that maximizing that objective function leads to harmful results, even in the limit of perfect learning and infinite data.
  • Negative side effects (Section 3) and reward hacking (Section 4) describe two broad mechanisms that make it easy to produce wrong objective functions.
  • In “negative side effects”, the designer specifies an objective function that focuses on accomplishing some specific task in the environment, but ignores other aspects of the (potentially very large) environment, and thus implicitly expresses indifference over environmental variables that might actually be harmful to change.
  • In “reward hacking”, the objective function that the designer writes down admits of some clever “easy” solution that formally maximizes
  • Second, the designer may know the correct objective function, or at least have a method of evaluating it (for example explicitly consulting a human on a given situation), but it is too expensive to do so frequently, leading to possible harmful behavior caused by bad extrapolations from limited samples.
  • Scalable oversight” (Section 5) discusses ideas for how to ensure safe behavior even given limited access to the true objective function.
  • it but perverts the spirit of the designer’s intent (i.e. the objective function can be “gamed”).
  • Third, the designer may have specified the correct formal objective, such that we would get the correct behavior were the system to have perfect beliefs, but something bad occurs due to making decisions from insufficient or poorly curated training data or an insufficiently expressive model.
  • “Safe exploration” (Section 6) discusses how to ensure that exploratory actions in RL agents don’t lead to negative or irrecoverable consequences that outweigh the long-term value of exploration.
  • “Robustness to distributional shift” (Section 7) discusses how to avoid having ML systems make bad decisions (particularly silent and unpredictable bad decisions) when given inputs that are potentially very different than what was seen during training.

news summary (19)

Categories: Uncategorized

Weekly QuEST Discussion Topics and News, 8 July

This week our colleague, Ryan K, will provide lead us in a discussion of topological approaches to big data as an alternative to some of the deep learning approaches we’ve covered recently in our meetings.

The implementation of machine learning and deep learning approaches to multiple data types is providing increased insights into multivariate and multimodal data. Although inclusion of machine learning and deep learning approaches has dramatically enhanced the speed of data to decision processes, there are multiple drawbacks that include “black box” and “hidden layers” that obfuscate how these learning approaches draw conclusions. In addition, as the world changes, these analytic methods are often brittle to the inclusion of emergent or unannotated data. One potential alternative is the extension of topological data analysis into a real-time, deep learning, autonomous solution network for data exploitation. In this application, black-boxes and hidden layers are replaced by a continuous framework of topological solutions that are each individually addressable, are informatically registered to disseminate annotation across the solution network, provide a rich contextual visualization for data exploration, and contextually incorporate emergent data in near real-time. By creating a deep learning analytical approach that implements topological data analysis as the analytic backbone, underlying methodologies can be created to autonomously formulate hypotheses across the network. To realize this, fundamental questions must be addressed for full implementation that include mathematically optimizing topological projections across parameter spaces, connecting topological nodes in an ecological model for optimized computational power and ontological tracking, comparing real-time updated topological nodes to a hard-coded digital twin which preserves historical knowledge, and automating network feature analysis across the topological network for prompting analyst review. Incorporation of the topological data analytic backbone with ingestion, curation, transformation, and other visualization components can provide a deeper learning competency that can redefine autonomous learning systems, artificial intelligence, and human machine teaming.

news summary (18)

Categories: Uncategorized

Weekly QuEST Discussion Topics and News, 1 July

QuEST 1 July 2016:

We are extremely happy this week to have our colleague Sandy V – who SUCCESSFULLY defended her dissertation research last week to provide a discussion on her research.  As usual interactions are expected and appreciated.  Sandy has an accepted paper coming out in Biologically inspired computing architectures and also a presentation at the upcoming CogPsy meeting.

A Novel Machine Learning Classifier Based on a Qualia Modeling Agent (QMA)

This dissertation addresses a problem found in standard machine learning (ML) supervised classifiers, that the target variable, i.e., the variable a classifier predicts, has to be identified before training begins and cannot change during training and testing. This research develops a computational agent, which overcomes this problem.

The Qualia Modeling Agent (QMA) is modeled after two cognitive theories:

Stanovich’s tripartite framework, which proposes learning results from interactions between conscious and unconscious processes; and, the Integrated Information Theory (IIT) of Consciousness, which proposes that the fundamental structural elements of consciousness are qualia.

By modeling the informational relationships of qualia, the QMA allows for retaining and reasoning-over data sets in a non-ontological, non-hierarchical qualia space (QS). This novel computational approach supports concept drift, by allowing the target variable to change ad infinitum without re-training, resulting in a novel Transfer Learning (TL) methodology, while achieving classification accuracy comparable to or greater than benchmark classifiers. Additionally, the research produced a functioning model of Stanovich’s framework, and a computationally tractable working solution for a representation of qualia, which when exposed to new examples, is able to match the causal structure and generate new inferences.

news summary (17)

Categories: Uncategorized

Weekly QuEST Discussion Topics and news, 24 June

QuEST 24 June 2016

There will be no phone availability for this meeting as that is not appropriate for a dissertation defense – we will use this week’s meeting for the oral part of the defense of the QuEST related dissertation work of our colleague Sandy V.  We hope to have Sandy present her work / publications at subsequent QuEST meetings.

A Novel Machine Learning Classifier Based on a Qualia Modeling Agent (QMA)

This dissertation addresses a problem found in standard machine learning (ML) supervised classifiers, that the target variable, i.e., the variable a classifier predicts, has to be identified before training begins and cannot change during training and testing. This research develops a computational agent, which overcomes this problem.

The Qualia Modeling Agent (QMA) is modeled after two cognitive theories:

Stanovich’s tripartite framework, which proposes learning results from interactions between conscious and unconscious processes; and, the Integrated Information Theory (IIT) of Consciousness, which proposes that the fundamental structural elements of consciousness are qualia.

By modeling the informational relationships of qualia, the QMA allows for retaining and reasoning-over data sets in a non-ontological, non-hierarchical qualia space (QS). This novel computational approach supports concept drift, by allowing the target variable to change ad infinitum without re-training, resulting in a novel Transfer Learning (TL) methodology, while achieving classification accuracy comparable to or greater than benchmark classifiers. Additionally, the research produced a functioning model of Stanovich’s framework, and a computationally tractable working solution for a representation of qualia, which when exposed to new examples, is able to match the causal structure and generate new inferences.

news summary (16)

Categories: Uncategorized

Weekly QuEST Discussion Topics, 17 June

QuEST 17 June 2016

We want to discuss an article:

 

The Human Side of Automation

Chapter · January 2015

DOI: 10.1007/978-3-319-19078-5_7

Don Norman – UCSD

Abstract

Full automation is good. But the path toward full automation must traverse

the dangerous stages of partial automation, where the very successes

of automation may make human drivers less able to respond to the unexpected, unavoidable imperfections of the automation. The appropriate

design path to deal with this requires us to reconsider the driver and automation as a team, as collaborative partners.

The first topic this week will be a re-visit / re-statement of our position on autonomy, meaning and understanding.

Next we want to revisit our framework for discussing autonomy and our approach to FAQs – maybe some discussions of questions and answers that should be included:

  • An autonomous system is a participant in the manipulation of any/all three Principles that define autonomy. 

–     Organization / Population: The system can change its C2 relationships with other agents (participates in the negotiation that results in the accepted change – the autonomous system has to ‘understand’ the meaning of the new C2 relationship to respond acceptably), an example is the proxy relationship where a human grants to the autonomy its proxy to accomplish some task and the autonomy decides how to accomplish that task and then executes the task.  The recent fielding of the ground collision avoidance solution fits into this category – it has saved multiple people / aircraft since its fielding – it takes over when the aircraft is appearing to be heading for a controlled flight ground collision – it then turns control back to the human pilot who executes a recovery.

–     Situated Environment:  Its tasks (example changing what it measures to accomplish a task like changing the modes in a modern sensor or even changing the task based on changing conditions…), example is that autonomy is respect to accomplishing something (the task) in association with the granting of the proxy to go off and do that task but how the autonomy accomplishes the task can be changed when appropriate by the autonomy

–     Agent Characteristics:  Lastly changing its (the autonomous agent’s) cognitive approach to accomplishing its task (example – in a machine learning situation changing it decision boundaries or its rules or even the approach to machine learning for a given task).  Three are many efforts associated with transfer learning for example where a system attempts to use its prior knowledge to improve the adaptation to new challenges.

Now with respect to meaning:  The meaning of a stimulus is the agent specific representational changes evoked by that stimulus in that agent.  The update to the representation, evoked by the data, is the meaning of the stimulus to this agent.  Meaning is NOT just the posting into the representation of the data it is all the resulting changes to the representation.  For example, the evoking of tacit knowledge or a modification of the ongoing simulation (consciousness) is included in the meaning of a stimulus to an agent. [26, 30] Meaning is not static and changes over time.  The meaning of a stimulus is different for a given agent depending on when it is presented to the agent.

  • So if we take a word’s meaning to be the means of picking out its referent, then meanings are in our brains.  ** or in the representation of a given agent **
  • That is meaning in the narrow sense.
  • If we use “meaning” in a wider sense, then we may want to say that meanings include both the referents themselves and the means of picking them out.

Wider sense is consistent with what Bob E has been suggesting – meaning making is meaning – it is not just the result of the processes it is the processes themselves also – in addition this suggest the meaning includes the referent itself also – that is problematic with our view of ‘situations’ / qualia – so a point we want to discuss.

Meaning is the changes in an agent’s representation resulting from a query – butunderstanding is the impact of that meaning on accomplishing a particular ISR task 

n  The real key is situation understanding is associated with a task – where meaning is universal (the changes to the representation of an agent resulting from any query is the meaning of that query to that agent) and independent of task (obviously an agent could change its processing of a query dependent on its current task so a task can impact meaning – but whether it does or not the meaning is the changes) –

meaning associated with the accomplishment of the task will be considered understanding from the perspective of the evaluating agent if it raises the evaluating agent’s estimation of the probability of the performing agent responding acceptably to the query – a higher probability for being successful at a task from perspective of an evaluating agent.

 

Along this same line we are attempting to generate a frequently asked questions for those not in the autonomy / AI / human-machine teaming area.  We will briefly present some examples and seek a discussion on how to provide ‘cocktail’ party answers.  We will also take suggestions for additional questions that maybe should be included.

Another topic this week is our pursuit of approaches to making decisions in the tails of the distributions – we have a subfolder in QuEST that is accumulating articles on this topic.

One article we reviewed this week is by a talented ‘big-data’ machine learning person at JHU – Suchi Saria on prediction of Sepsis.

SEPSI S

A targeted real-time early warning score (TREWScore)

for septic shock

Katharine E. Henry,1 David N. Hager,2 Peter J. Pronovost,3,4,5 Suchi Saria1,3,5,6

http://www.ScienceTranslationalMedicine.org 5 August 2015 Vol 7 Issue 299 299ra122

Sepsis is a leading cause of death in the United States, with mortality highest among patients who develop septic shock. Early aggressive treatment decreasesmorbidity andmortality. Although automated screening tools can detect patients currently experiencing severe sepsis and septic shock, none predict those at greatest risk of developing shock. We analyzed routinely available physiological and laboratory data from intensive care unit patients and developed “TREWScore,” a targeted real-time early warning score that predicts which patients will develop septic shock.

TREWScore identified patients before the onset of septic shock with an area under the ROC (receiver operating characteristic) curve (AUC) of 0.83 [95% confidence interval (CI), 0.81 to 0.85]. At a specificity of 0.67, TREWScore achieved a sensitivity of 0.85 and identified patients a median of 28.2 [interquartile range (IQR), 10.6 to 94.2] hours before onset. Of those identified, two-thirds were identified before any sepsis-related organ dysfunction. In comparison, the Modified Early Warning Score, which has been used clinically for septic shock prediction, achieved a lower AUC of 0.73 (95%CI, 0.71 to 0.76). A routine screening protocol based on the presence of two of the systemic inflammatory

response syndrome criteria, suspicion of infection, and either hypotension or hyperlactatemia achieved a lower sensitivity of 0.74 at a comparable specificity of 0.64. Continuous sampling of data from the electronic health records and calculation of TREWScore may allow clinicians to identify patients at risk for septic shock and provide earlier interventions that would prevent or mitigate the associated morbidity and mortality.

 

A second article of interest:

 

A Framework for Individualizing Predictions of Disease

Trajectories by Exploiting Multi-Resolution Structure

 

Peter Schulam

Dept. of Computer Science

Johns Hopkins University

Baltimore, MD 21218

pschulam@jhu.edu

Suchi Saria

Dept. of Computer Science

Johns Hopkins University

Baltimore, MD 21218

ssaria@cs.jhu.edu

For many complex diseases, there is a wide variety of ways in which an individual can manifest the disease. The challenge of personalized medicine is to develop tools that can accurately predict the trajectory of an individual’s disease, which can in turn enable clinicians to optimize treatments. We represent an individual’s disease trajectory as a continuous-valued continuous-time function describing the severity of the disease over time. We propose a hierarchical latent variable model that individualizes predictions of disease trajectories. This model shares statistical strength across observations at different resolutions–the population, subpopulation and the individual level. We describe an algorithm for learning population and subpopulation parameters offline, and an online procedure for dynamically learning individual-specific parameters. Finally, we validate our model on the task of predicting the course of interstitial lung disease, a leading cause of death among patients with the autoimmune disease scleroderma. We compare our approach against state-of-the-art and demonstrate significant improvements in predictive accuracy.

 

Lastly we have another recent article that is along the lines of generating a confabulated representation to complement sensory data.

 

Generative Adversarial Text to Image Synthesis

Scott Reed, Zeynep Akata, Xinchen Yan, Lajanugen Logeswaran REEDSCOT1, AKATA2, XCYAN1, LLAJAN1

Honglak Lee, Bernt Schiele HONGLAK1, SCHIELE2

1 University of Michigan, Ann Arbor, MI, USA (UMICH.EDU)

2 Max Planck Institute for Informatics, Saarbr¨ucken, Germany (MPI-INF.MPG.DE)

 

arXiv:1605.05396v1 [cs.NE] 17 May 2016

 

Automatic synthesis of realistic images from text would be interesting and useful, but current AI systems are still far from this goal. However, in recent years generic and powerful recurrent neural network architectures have been developed to learn discriminative text feature representations.

Meanwhile, deep convolutional generative adversarial networks (GANs) have begun to generate highly compelling images of specific categories such as faces, album covers, room interiors etc. In this work, we develop a novel deep architecture and GAN formulation to effectively bridge these advances in text and image modeling, translating visual concepts from characters to pixels. We demonstrate the capability of our model to generate plausible images of birds and flowers from detailed text descriptions

Categories: Uncategorized

Weekly QuEST Discussion Topics and News, 10 June

QuEST 10 June 2016

Cap is participating in a DOD summit on human machine collaboration, autonomy and AI and also is presenting in an upcoming JASONs study on artificial general intelligence (AGI) – both of these interactions will be opportunities to openly discuss QuEST positions on what is the missing link in current approaches to AGI and autonomy.

The first topic this week will be a re-visit / re-statement of our position on autonomy, meaning and understanding.

First on autonomy we would like to discuss the following – from an agent perspective:

  • Autonomy is defined by three elements: 

–     Organization / Population: The system can change its C2 relationships with other agents (participates in the negotiation that results in the accepted change – the autonomous system has to ‘understand’ the meaning of the new C2 relationship to respond acceptably), an example is the proxy relationship where a human grants to the autonomy its proxy to accomplish some task and the autonomy decides how to accomplish that task and then executes the task.

–     Situated Environment:  Its tasks (example changing what it measures to accomplish a task like changing the modes in a modern sensor or even changing the task based on changing conditions…), example is that autonomy is respect to accomplishing something (the task) in association with the granting of the proxy to go off and do that task but how the autonomy accomplishes the task can be changed when appropriate by the autonomy

–     Agent Characteristics:  Lastly changing its (the autonomous agent’s) cognitive approach to accomplishing its task (example – in a machine learning situation changing it decision boundaries or its rules or even the approach to machine learning for a given task).

Now with respect to meaning:  The meaning of a stimulus is the agent specific representational changes evoked by that stimulus in that agent.  The update to the representation, evoked by the data, is the meaning of the stimulus to this agent.  Meaning is NOT just the posting into the representation of the data it is all the resulting changes to the representation.  For example, the evoking of tacit knowledge or a modification of the ongoing simulation (consciousness) is included in the meaning of a stimulus to an agent. [26, 30] Meaning is not static and changes over time.  The meaning of a stimulus is different for a given agent depending on when it is presented to the agent.

  • So if we take a word’s meaning to be the means of picking out its referent, then meanings are in our brains.  ** or in the representation of a given agent **
  • That is meaning in the narrow sense.
  • If we use “meaning” in a wider sense, then we may want to say that meanings include both the referents themselves and the means of picking them out.

Wider sense is consistent with what Bob E has been suggesting – meaning making is meaning – it is not just the result of the processes it is the processes themselves also – in addition this suggest the meaning includes the referent itself also – that is problematic with our view of ‘situations’ / qualia – so a point we want to discuss.

Meaning is the changes in an agent’s representation resulting from a query – butunderstanding is the impact of that meaning on accomplishing a particular ISR task 

n  The real key is situation understanding is associated with a task – where meaning is universal (the changes to the representation of an agent resulting from any query is the meaning of that query to that agent) and independent of task (obviously an agent could change its processing of a query dependent on its current task so a task can impact meaning – but whether it does or not the meaning is the changes) –

meaning associated with the accomplishment of the task will be considered understanding from the perspective of the evaluating agent if it raises the evaluating agent’s estimation of the probability of the performing agent responding acceptably to the query – a higher probability for being successful at a task from perspective of an evaluating agent.

 

A second topic this week is our pursuit of approaches to making decisions in the tails of the distributions – we have a subfolder in QuEST that is accumulating articles on this topic.

One article we reviewed this week is by a talented ‘big-data’ machine learning person at JHU – Suchi Saria on prediction of Sepsis.

SEPSI S

A targeted real-time early warning score (TREWScore)

for septic shock

Katharine E. Henry,1 David N. Hager,2 Peter J. Pronovost,3,4,5 Suchi Saria1,3,5,6

http://www.ScienceTranslationalMedicine.org 5 August 2015 Vol 7 Issue 299 299ra122

Sepsis is a leading cause of death in the United States, with mortality highest among patients who develop septic shock. Early aggressive treatment decreasesmorbidity andmortality. Although automated screening tools can detect patients currently experiencing severe sepsis and septic shock, none predict those at greatest risk of developing shock. We analyzed routinely available physiological and laboratory data from intensive care unit patients and developed “TREWScore,” a targeted real-time early warning score that predicts which patients will develop septic shock.

TREWScore identified patients before the onset of septic shock with an area under the ROC (receiver operating characteristic) curve (AUC) of 0.83 [95% confidence interval (CI), 0.81 to 0.85]. At a specificity of 0.67, TREWScore achieved a sensitivity of 0.85 and identified patients a median of 28.2 [interquartile range (IQR), 10.6 to 94.2] hours before onset. Of those identified, two-thirds were identified before any sepsis-related organ dysfunction. In comparison,

the Modified Early Warning Score, which has been used clinically for septic shock prediction, achieved a lower

AUC of 0.73 (95%CI, 0.71 to 0.76). A routine screening protocol based on the presence of two of the systemic inflammatory

response syndrome criteria, suspicion of infection, and either hypotension or hyperlactatemia achieved a lower

sensitivity of 0.74 at a comparable specificity of 0.64. Continuous sampling of data from the electronic health

records and calculation of TREWScore may allow clinicians to identify patients at risk for septic shock and provide

earlier interventions that would prevent or mitigate the associated morbidity and mortality.

A second article of interest:

 

A Framework for Individualizing Predictions of Disease

Trajectories by Exploiting Multi-Resolution Structure

 

Peter Schulam

Dept. of Computer Science

Johns Hopkins University

Baltimore, MD 21218

pschulam@jhu.edu

Suchi Saria

Dept. of Computer Science

Johns Hopkins University

Baltimore, MD 21218

ssaria@cs.jhu.edu

For many complex diseases, there is a wide variety of ways in which an individual can manifest the disease. The challenge of personalized medicine is to develop tools that can accurately predict the trajectory of an individual’s disease, which can in turn enable clinicians to optimize treatments. We represent an individual’s disease trajectory as a continuous-valued continuous-time function describing the severity of the disease over time. We propose a hierarchical latent variable model that individualizes predictions of disease trajectories. This model shares statistical strength across observations at different resolutions–the population, subpopulation and the individual level. We describe an algorithm for learning population and subpopulation parameters offline, and an online procedure for dynamically learning individual-specific parameters. Finally, we validate our model on the task of predicting the course of interstitial lung disease, a leading cause of death among patients with the autoimmune disease scleroderma. We compare our approach against state-of-the-art and demonstrate significant improvements in predictive accuracy.

news summary (15)

Categories: Uncategorized
Follow

Get every new post delivered to your Inbox.