Weekly QuEST Discussion Topics and News, 20 Nov

November 19, 2015 Leave a comment

QuEST this week will focus on discussing the Andres questions about unexpected query:


What is an unexpected Query v2?


Nov 2015


What is an agent?


  • We will use the term agent in a broad manner – to include human and computer agents
  • The common functionality of agents is they have sensors that allow them to capturestimuli from the environment
  • Once the stimuli are captured by an agent’s sensors and brought into the agent that is what we call data (note you cannot say data without saying with respect to which agent)
  • The agent then updates some aspect of its internal representation with that data thus reducing some uncertainty in that representation – so we now call that information(again note that information is agent centric as is data)
  • The agent might generate some action using its effectors or the effectors might just output stimuli into the environment for use by other agents or the agent might just update its internal representation in response to the information obtained.



What is a query?


  • Let’s define Query – as the act of a stimulus being provided to an agent – the stimulus has characteristics that completely capture the salient axes (keep in mind what is salient in a stimulus is agent centric) of the stimuli – some of those axes are captured by an agent in its conversion of that stimuli into data (agent centric internal representation of the stimuli)
  • We use the term query instead of stimuli to capture the idea that a given agent must take the stimuli and appropriately respond (thus an action)

–     that response may be to just update its representation or not or may actually be taking an action through an agent’ effectors


What is an unexpected query?


  • That is the goal of this discussion to bring some specifics to the idea of anunexpected query – but for now realize we use that term with some localization (maybe with respect to an agent – or maybe with respect to some process within an agent = performing agent – that is an unexpected query could be unexpected to some process within an agent or to the agent or collection of agents as a whole) – it is deemedunexpected to the performing agent and that label ‘unexpected’ is from the perspective of an external ‘evaluating agent’

–     But the point of the word ‘unexpected’ is to capture the idea that a process that takes in stimuli and responds (again could just be updating a representation or could be the response is some action taken) has some assumptions built into its design (from the perspective of an evaluating agent) that may or may not be violated by a given stimulus (and the violation is from the perspective of an external agent)

–     An example might help here – if I know you are educated to be a Dentist but then I (as the evaluating agent) observe you having to respond to a medical emergency, from my perspective (as the evaluating agent) I would classify the stimuli (having to respond to a medical emergency) as an unexpected query to you (the Dentist) in the sense that from my perspective I have no confidence you will perform acceptably to this stimuli

  • When the assumptions that are key for the acceptable response are violated we will term that stimuli as being unexpected to that process/agent (performing agent) – and the violation is from the perspective of the evaluating agent(specifically from the evaluating agent’s understanding of the performing agent’s preparedness to respond acceptably for a given stimulus).

–     Another example might help here – from the ideas associated with transfer learning see for example the survey article by Pan/Yang – using a machine learning approach I design a solution within a Domain (feature space and probability distribution), one of their examples is document classification, even within the domain of document classification if I as the evaluating agent note that the features are different OR the probability distributions have changed from how the system was trained then I would have little confidence that the system would respond acceptably and thus I (as the evaluating agent would deem the query unexpected).  Note how the change in features is related to the awareness of the agent to the environment and the change in the probability distribution is associated with the derived model the machine learning agent is using to respond to the query.

  • Note – the ‘unacceptable’ nature of any response is determined from the perspective of some other agent {evaluating agent} / process

–     so where one evaluating agent might take the response to a stimuli as unacceptable another may deem it perfectly acceptable – thus the unexpected nature of a stimuli is agent centric (note an agent – the evaluating agent – different than the one reacting to the stimulus the performing agent)

–     Note the evaluating agent has access to additional stimuli AND also has a model of the performing agent – thus can assess that the performing agent has an unacceptable response from its perspective


So why do we care:


  • So we posit that the type 2 processes (consciousness results from type 2 processing) that are situated and simulation based and include a model for the type 1 processes (that is what intuition provides us – at the conscious level it is our model of the evaluation of the Type 1 system projected into a conscious form) can be evaluation agents and detect UQs for the Type 1 processes
  • I suspect similarly the Type 1 processes might receive inputs from Type 2 stimuli and can possibly update their models – a late set of interactions with colleagues Robert P / Mike Y convince me this is a defendable position
  • Bottom line – a simulation based / situated representation can generate solutions to queries where the solutions don’t have to based on the stimuli – they are based on inferences from the simulation/ situation based representation – and from the evaluation agent called evolution perspective that provided more acceptable responses in leading to more reproduction …


What is consciousness?


  • Stable, consistent and useful situated simulation that is structurally coherent

–     The space for the unexpected query for such a representation complements the experiential representation which is the focus of Type 1 processes – so it is the QuEST position that consciousness provides an ability to reduce the space of unexpected queries for agents with only experiential based Type 1 processes

From our chapter on situation consciousness for cyber autonomy:

















What is an unexpected Query to AC3 – AFRL conscious content curation?


How can our implementation of CNN/RNN/LSTM/CLSTMs be used to get an acceptable response when we push out of the upper right quadrant of the UQ space discussed above.  First let’s examine reduction in awareness (moving to the left on the horizontal axis.  For example how can our approach be more robust to sensor errors?  Imagine pixels going bad in the video sensor generating the frame sequence so there are blank spots / blind spots in the field of view.  (now this may seem far-fetched – but where I’m going is even when we get clean measurements our exploitation of that data may be incorrect so generating an imagined representation where content is inferred versus evoked by sensory data would be analogous to the system as if pixels had gone bad).  So again how do we use the vocabulary of our word/thought vectors to infer other location based thought vectors that might be different than the ones that got evoked associated with that location by the bottom up sensory data processing.  This would be an example of moving to the left in the graphs above.


Similarly imagine the issue of having to combine models to form new models that did not exist when the system was trained.  In some sense the transfer learning we accomplish by using the image net training set for the CNN then refining the thought vectors with some video data is an example.  The video data is an UQ to the original image net trained solution.  We’ve moved down the vertical axis.  We have to enter a retraining period to get to the situation of making the system respond acceptable to video snippet stimuli from our military sensors.  Another example would be attempting to use other learned concepts and do conceptual combination to form completely new concepts.


Cognition is inherently constructive.

–     Our cognitive functioning is not confined to retrieving familiar ideas and concepts, but rather is predicated upon the ability to understand new things and represent new concepts. *** unexpected Query ***


Conceptual combination research investigates the processes involved in creating and understanding new meanings from old referents.

–     For example, how do people interpret novel combinations such as cactus beetle, mouse potato, or fame advantage? ** compound Qualia  – Pencil thin woman**


Such combinations are used liberally (conversations, newspaper headlines, signage, novels, etc.) and people generally have little difficulty in constructing plausible interpretations, even where the surrounding context may be quite limited or uninformative.


How can we use our AC3 infrastructure to achieve a constructive cognition engine?


Cognition is inherently constructive –  ** I would contend sensing is inherently constructive  so the same processes can be use for concept combination and perception**


When a malware author combines sequences of actions / commands in novel ways they are generating a conceptual combination that previously has NOT been experienced is NOT a familiar idea in the sense of what is stored in memory – thus will require the detection system to use approaches to conceptual combination to determine an acceptable ‘meaning’ of the new code / action sequence


Similarly when the operating conditions change in a sensing task – this requires a solution to conceptual combination to determine the meaning of the new (previously not experienced) combination of observations

news summary (33)

Categories: Uncategorized

Weekly QuEST Discussion Topics and News, 13 Nov

November 12, 2015 Leave a comment

QuEST 13 Nov 2015

This week we will first continue our discussion of big data and QuEST and Deep learning – we will review the information from the guest lecture by Prof Mixon

He gave the local QuEST guys a lecture: Recent advances in mathematical data science

This talk describes three different problems of interest in mathematical data science. First, we consider the problem of compressive classification, in which one seeks to classify data from compressed measurements. Next, we consider a semidefinite relaxation of the k-means problem to establish clustering guarantees under a certain data model. Finally, we recast the general problem of binary classification as a sparse approximation problem, and we observe numerical evidence of the efficacy of this approach

Next we will discuss the couple of slides we need to add on the topic of Big Data to our ‘Kabrisky lecture’ – “What is QuEST?’ deck on the topic – we give the Kabrisky Lecture the first meeting of the calendar year – it will 8 Jan 2016 at noon eastern.

After that discussion on what slides to add that summarize QuEST and big Data / deep learning – I want to spend a couple of minutes talking about an effort that came out of the naval post graduate school (NPS) – “Global Information Network Archtecture = GINA” – our colleagues that work in this area spent some time while I was out last week adapting their efforts to our QuEST terminology – I will briefly attempt to familiarize the group with the effort and post some GINA material on the internal QuEST site for those interested in pursuing it.

Next we want to continue our review of the material we’ve covered this calendar year for possible inclusion into the Kabrisky Lecture slide deck – one such topic is ‘meaning’ – so we will spend some time discussing what we’ve concluded about ‘meaning’ and what couple of slides capture the key ideas for inclusion.

news summary (32)

Categories: Uncategorized

Weekly QuEST Discussion Topics, 6 Nov

November 6, 2015 Leave a comment

QuEST 6 Nov 2015


This week we will continue our discussion of big data and QuEST – and also Deep learning – we will have a guest lecture by Prof Mixon from AFIT – to accommodate his teaching commitments we have to change the location of the meeting to – room 102 in Bldg 646 for QuEST 1200-1300. There is no phone in that auditorium.  So unfortunately for our colleagues away from Wright Patterson we will have to provide an after action update




Recent advances in mathematical data science



This talk describes three different problems of interest in mathematical data science. First, we consider the problem of compressive classification, in which one seeks to classify data from compressed measurements. Next, we consider a semidefinite relaxation of the k-means problem to establish clustering guarantees under a certain data model. Finally, we recast the general problem of binary classification as a sparse approximation problem, and we observe numerical evidence of the efficacy of this approach

Categories: Uncategorized

Weekly QuEST Discussion Topics and News, 30 Oct

October 28, 2015 Leave a comment

QuEST 30 Oct 2015

First we want to discuss a recent article:

An ISR Perspective on Fusion Warfare

Maj Gen VeraLinn “Dash” Jamieson, USAF

Lt Col Maurizio “Mo” Calabrese, USAF

… The Air Force’s mission is to fly, fight, and win today and tomorrow’s wars. How we

accomplish this must undergo a paradigm shift. This change is an imperative not only

for the Air Force, but also for all the US armed services and elements of the Intelligence

Community (IC). A review of open source literature on fifth generation weapon systems

(e.g., B-2A Spirit, F-22 Raptor, F-35 Lightning II) presents a common theme:

near real-time information sharing on threats, targets, onboard payloads, aircraft

flight dynamics, and command and control (C2) activities.1

The pilots in those stealth platforms act as the central nervous system in the cockpit

to integrate disparate types of data and make decisions. Simultaneously on the ground,

intelligence, surveillance, and reconnaissance (ISR) Airmen serve as the central

nerve center in intelligence squadrons to process information coming in from airborne,

space, cyber, and terrestrial collection sources. these Airmen must fuse multiple

types of data from numerous sources in a fast-paced environment to produce analysis

and empower decision makers at various Air Force command and control nodes.

These C2 nodes may include air and space operations centers (AOCs), the

common ground system (CGS) core sites, unit-level intelligence flights, or even the

pilot in the cockpit. Harnessing the information available from each of these elements

in a coherent, collaborative, and cohesive manner will provide decision advantage and

success in tomorrow’s conflicts. This paper defines and explores a concept we call “fusion

warfare” — and provides a perspective on what tomorrow’s warfighting will mean

for Air Force ISR professionals.

In future conflicts, the victor may not necessarily be the one with the quickest OODA

Loop. Rather, the prevailing side may be the one which can harness the power of multiple

OODA loops, utilize the vast amounts of data in them, and provide enhanced

battlefield situational awareness—all fused into decision-making analysis—to achieve

multi-domain freedom of action.

One of the concepts related to QuEST in this discussion is big data – this time of year we spend bandwidth reviewing what we’ve discussed this CY in QuEST and attempt to integrate the material in our ‘What is QuEST briefing’ – we give the first meeting of a new CY.  Clearly this year we’ve spent lots of time talking about Deep learning and lots of time talking about big-data.  So from the perspective of Fusion Warfare – we want to have  discussion how to capture for our What is QuEST deck of slide our position on use of Deep learning – what is the relationship between big data and QuEST – what is the relationship between big data and Deep learning – and how do we capture these positions in a couple of slides / talking points.

news summary (31)

Categories: Uncategorized

Weekly QuEST Discussion Topics and News, 23 Oct

October 22, 2015 Leave a comment

QuEST 23 Oct 2015

We want to start this week talking about ISR challenges with respect to big Data (the discussions we’ve been having over the last several weeks on big-data applied to the issues we face in the ISR enterprise – we want to map where big data and QuEST will impact C4ISR ) our colleague Prof O has an opportunity on Monday to speak to some potential international collaborators so we will take the opportunity to refine our message on the QuEST perspectives on big data specifically for C4ISR

Next we also want to return to the discussion on pain – we didn’t make it to some points I still want to discuss – for example how do we truth human states that are below the level of consciousness – but the results are posted consciously – like pain and emotion – we will post a couple of articles on estimating emotion

Neural Network-Based Improvement in Class

Separation of Physiological Signals for Emotion


E. Leon, G. Clarke, F. Sepulveda, V. Callaghan

Department of Computer Science, University of Essex

Colchester, Essex, UK.

eeleon@essex.ac.uk, graham@essex.ac.uk, fsepulv@essex.ac.uk, vic@essex.ac.uk

Abstract—Computer scientists have been slow to become

aware of the importance of emotion on human decisions and

actions. Recently, however, a considerable amount of research

has focused on the utilisation of affective information with the

intention of improving both human-machine interaction and

artificial human-like inference models. It has been argued that

valuable information could be obtained by analysing the way

affective states and environment interact and affect human

behaviour. A method to improve pattern recognition among four

bodily parameters employed for emotion recognition is

presented. The utilisation of Autoassociative Neural Networks

has proved to be a valuable mechanism to increase inter-cluster

separation related to emotional polarity (positive or negative). It

is suggested that the proposed methodology could improve

performance in pattern recognition tasks involving physiological

signals. Also, by way of grounding the immediate aims of our

research, and providing an insight into the direction of our work,

we provide a brief overview of an intelligent-dormitory test bed

in which affective computing methods will be applied and

compared to non-affective agents.

Optimised Attribute Selection for Emotion classification using Physiological Signals

Leon, Clarke, Sepulveda, Callaghan

news summary (30)

Categories: Uncategorized

Weekly QuEST Discussion Topics and News, Oct 16

October 15, 2015 Leave a comment

QuEST 16 Oct 2015:

We want to start this week with a discussion associated with big-data. Specifically how does big-data relate to approaches like deep learning and then how will big-data change our situation awareness for decision making. Capt Amerika was asked this week for an interview on the subject and we will spend a little time discussing the points he made – all related to QuEST.

The second topic is associated with Pain. Specifically Adam R has been working in the area of prediction of performance (exercise/training etc) from physiological (things like heart rate variability) and consciously reported readiness (things like how much sleep did you get?, how do you feel?) and specifically how to tailor exercise/training appropriately to maximize results. That led to a discussion on the concept of pain. See for example


first the big data interview information:

Question to 88th ABW from a reporter: As we discussed, I’m writing an article on “Big Data on the Battlefield” for Defense Systems magazine. I’d like to interview Dr. Steve “Cap” Rogers, a senior scientist at AFRL and the Air Force’s principle scientific authority for automatic target recognition and sensor fusion.

Reporter: I’ll be covering how big data and analytics are improving situational

awareness, target acquisition, etc.

Cap comments in Carolina blue:

Entering comments:

Background: breakthroughs in computers have given us the tools to store and manipulate data at scales we never dreamed of, but the computers have to be explicitly instructed how to derive value from data.

Example: Does this pattern of activity imply a force build up? The machine can find patterns and possibly even note unusual patterns (assuming it has been given enough data to establish a model of what is normal) but must be then programmed to associate those patterns with some semantic interpretation and that is provided by human operators (example – when you see this pattern alert the operator that a missile is about to launch).

The problem is without the ability to understand what the semantic interpretation of the data is the machine often can’t generalize to irrelevant variations in the observations.

There is a school of thought that with enough data irrelevant variations of the data can be learned by the big data analysis so this may become less a problem. (see for example The Unreasonable Effectiveness of Data – by Alon Halevy, Peter Norvig, and Fernando Pereira, of Google)

Important conclusion – the meaning of the observations to the machine is NOT the meaning of the observations to the human operator and can lead to really bad decisions.

Therefore any attempt to via machines to characterize the environment currently rely on this explicit coding of what to do with what measurements and the computer results are then provided so the human extracts the meaning to make decisions

Restating the important conclusion: big data detects patterns in large data sets (often very valuable patterns in data sets with low information density {low SNR}) but most often doesn’t ask why (not designed to determine meaning in a manner consistent with human meaning making).

I am not a believer that big data can be used independent of theory and models

Although some of the results demonstrated using big data approaches are extremely impressive/accurate the ideas that

we don’t have to worry about statistical sampling/bias or

correlation at scale can always replace causation or

we don’t need to worry about the development of fundamentals based models

are all overly simplistic and can be very misleading.

see for example:


March 28, 2014: Big data: are we making a big mistake? By Tim Harford:

Accuracy achieved using big data may lead to confident decision making. Better decisions can mean greater operational efficiency, more cost reduction and reduced risk so I’m not saying there is no value to big data solutions.

Big data infrastructure is however a solved problem (from the IQT (In-Q-tel) quarterly (vol7 no 2) fall 2015 issue – discusses “Artificial Intelligence gets Real”.

Predictions with Big Data By Devavrat Shah:

We know how to collect massive amounts of data (e.g., web scraping, social media, mobile phones),

how to store it efficiently to enable queries at scale (e.g., Hadoop File System, Cassandra) and

how to perform computation (analytics) at scale with it (e.g., Hadoop, MapReduce).

And we can sometimes visualize it (e.g., New York Times visualizations).

However, we are unable to make meaningful decisions using the data unless those decisions are ones that we’ve preprogramed in what to reflexively do for a given environmental state determined by an understood co-occurrence of some observables – big data currently can’t handle the ‘unexpected query’. Actually it is more appropriate to say big data solutions are terrible at unexpected queries – in the sense they can be very misleading because they don’t understand the implications of their conclusions.

One classic example of big data bad conclusions – Black Swan issue:

http ://edge.org/conversation/the-fourth-quadrant-a-map-of-the-limits-of-statistics

… IF you are trying to make a decision in an environment with complex payoffs where there might be “black swans, (Black Swans, the highly improbable and unpredictable events that have massive impactTaleb – initially applied to financial markets extended to WWI, Soviet Union falling, 9-11 attacks, …) DO NOT rely on statistical methods (big data) — they will fail miserably at some point and notions like standard deviation and variance are nearly meaningless.

Turkey example — every day for the first 1,000 day of its life, a turkey might accumulate evidence that humans care about its welfare and it

might estimate very high confidence in the validity of its models (big data) — but the 1,001st day does not go so well. …

Or from Pentland: With Big Data you can easily get false correlations, for instance, “On Mondays, people who drive to work are more likely to get the flu.” If you look at the data using traditional methods, that may actually be true, but the problem is why is it true? Is it causal? Is it just an accident?

You won’t know. Normal big data analysis methods won’t suffice to answer those questions. What we have to come up with is new ways to test the causality of connections in the real world far more than we have ever had to do before. We no can no longer rely on laboratory experiments; we need to actually do the experiments in the real world.

Reporter Questions:

Is the challenge gathering big data or combining it and analyzing it quickly enough for it to be valuable?

The world has changed completely – in the 1960s we were happy to have the Brown Corpus for those of us working in natural language understanding – a million words – we now have Google releasing trillion word data sets (a million times larger) – so as I said in the entry remarks – we have no problem gathering data – (see the unreasonable effectiveness of data Google article)

In the military for example in the last decade – data from UAS went from 5 megabytes/sec to 3k megabytes/sec (3 gbytes/sec)

At the recent AFA conference the DCS for ISR stated in 2014 the DCGS weapon system processed 445k hours of FMV

If you look at our ability with systems like Gorgon Stare to observe persistently over tens of square Kms – you see the issue is NOT gathering the data.

We used to in the commercial world (and in some cases today) spend lots of effort wrangling the data – get rid of bad data … – due to the fragility of some of the tools – but there are examples where the data set is so large that we can extract the ‘model’ of the world the data provides and the model can represent even the very rare things that might occur (including errors like misspellings etc that we used to attempt to eliminate) – great example is speech recognition or language translation – but for applications where we need meaning we still have issues – things like document classification or sentiment analysis are still

challenging in the commercial world

In ISR the big data challenges are many (examples below from Jon “Doc” Kimmanau) I concur with his parsing and have edited some of the contents below:

Intelligence discovery is the ability to select, manipulate, and correlate data from multiple sources in order to identify information relevant to ongoing operations and requirements – here if we can pre-model a measure of ‘relevance’ we have some shot with existing tools – we seek discovery to be more robust and less robotic.

Intelligence assessment is the ability to provide focused examination of data and information about an object or an event, to classify and categorize it, and to assess its reliability and credibility in order to create estimates of capabilities and impact – we have a very general definition of event here to be anything that an operator needs to act upon.

Intelligence explanation is the ability to examine events and derive knowledge and insights from interrelated data in order to create causal descriptions and propose significance in greater contexts.this is at the crux of your question – so yes this is a challenge but only one of the challenges and although not in this parsing – the timeliness of the explanation has to achieve mission objectives as you suggest in your question.

Intelligence anticipation is the ability to warn and describe future states of the environment based on the manipulation and synthesis of past and present data again here we are good at this if we have in our experience set exactly what we are observing and thus be able to predict what should happen next we are very poor at machine based imagined futures that we haven’t programmed into them the possibility of this to occur.

Intelligence delivery is the ability to develop, tailor, and present intelligence products and services according to customer requirements and preferences – currently we are basically factory line generators of

intelligence products versus generating representations that are useable by many different war fighters for a variety of purposes – an idea we’ve called sensing as a service.

With respect to the question about making sense of the data quickly – Trend is absolutely streaming analytics – making sense of what we are observing while we are observing it – change what we are observing as a result – and how we are processing it – streaming analytics – so you change what you capture and how you process it getting out of the vacuum mentality that assumes some subsequent processing can find good information – and keep in mind when I say making sense here I’m including the perspective that currently I have to program into the machine what to do with the data and thus what the machine might conclude as ‘making sense’ all of that meaning is programmed responses associated with knowledge by the designing engineer of what questions the system needs to respond to – expected queries.

Are the capabilities involved with automated analysiscombining unstructured data from different platforms (text data, pictures, videos, sensor readings) and being able to combine it into a sensible form (overlaid on a map)—currently available?

Only in limited well defined applications have we been able to in an automated way combine diversity of data types and overlay in a ‘sensible form’ we have to program in the meaning (sensible form) too often we see analyst doing these cut/paste/overlays versus analysis (where the sensemaking occurs) – so we are as quickly as possible making strides to automate these sorts of thingsyour example with respect to overlays on maps fits that mode

We’ve recently found that event based representations are often more valuable than map based – but the real issue is how to formulate information in a manner that together with the human analyst accomplish some mission (sometimes it is object based, sometimes it is activity based but in general we need an approach that can accommodate forms for this operator doing this task at this moment with this data) – overlays are common but we are actively attacking the issue of not just using overlays because although they accelerate a current approach to analysis we are seeking to find what approach (example event based representation / processing) provides greater mission capability

that may or may not be fusing diverse sources/types of information into a map display

Much data today is not natively in structured format – so the premise of your question ‘unstructured’ is appropriate; as an example in the commercial world some data like tweets and blogs are weakly structured pieces of text, and some data like images and video are structured for storage and display, but not for semantic content / search – to do our missions we need to understand the semantic content and be able to search based on the meaning of that information – we often call that meta-data – we currently face the challenge of the automatic generation of semantic meta-data: transforming such content into a structured format for later analysis is a major challenge – in the QuEST discussion below we speak of a current effort to address this.

In the words of Jeff Jonas (IBM fellow) – The value of data explodes when it can be linked with other data, thus data integration is a major creator of value. We have the challenge to facilitate linkage during capture and to automatically link previously created data. Again the premise of your question is valid – Data analysis is a clear bottleneck in many applications, both due to lack of scalability of the underlying algorithms and due to the complexity of the data that needs to be analyzed.

Is the analysis of that data still best achieved through people rather than


The question is a little misleading – people are the only way to make meaning –but we are making great strides in facilitating more bandwidth for the human through automation of specific well define processing steps to facilitate humans doing analysis machines can only process in an ‘autisticmanner discussed above – do exactly what they are programed to do – these machine actions can be quite complex but have to be well understood / modeled – they don’t understand their recommendations in any meaningful way

When Target (retail store) (example from the big data big mistake article) big data analysis sends a home information associated with products useful when you are pregnant (baby clothes and maternity wear) they have no understanding of the implications (teenage daughter in house and a mother beyond child bearing years – they only noticed the correlation to purchase of unscented wipes and magnesium supplements) – and although accurate there is little analysis of false positive issues (the

big data analysis only captures that coupons dispensed in this fashion get used more often thus it is a commercially valuable approach to concluding to send coupons {decision} when these observables exist – the meaning of this correlation to the computer is nothing more than a calculation of the commercial return on this decision – there are many false positives in its calculation – but the commercial tradeoff between the true positives and the false positives is the source of the decision – the meaning of being pregnant to the humans involved in the transaction is quite different)

When google flu (The Parable of Google Flu: Traps in Big Data Analysis David Lazer, Ryan Kennedy, Gary King, Alessandro Vespignani) made many errors (doubling the flu numbers incorrectly) it didn’t understand why it was wrong

Or one more example – Similarly the recent Google story: http://www.foxnews.com/tech/2015/07/02/google-apologizes-for-misidentifying-black-couple-as-gorillas-in-photos-app/

 By Enid Burns Published July 02, 2015

As impressive as facial recognition software is, it isn’t perfect, and sometimes algorithm errors can lead to unintended consequences. Google discovered this in what’s possibly one of the worst scenarios a company could face: Its new Photos app incorrectly labeled a black couple as “gorillas.” This is a classic big data example – the meaning to the computer of gorilla is a combination of pixel intensities.

From colleagues Laurie F / Robert P:

although advances in technology are working toward improving the capability of the intelligence community to transform data collected from multiple sources into actionable intelligence (e.g., processing of ‘big data’ by deep learning algorithms; object detection and tracking by machine vision; integration and presentation of multi-INT data on novel information displays),

in the end it is the intelligence analyst—the human—who must understand and make meaning of the material being presented.

What is the current status of Qualia Exploitation of Sensor Technology, or QuEST, which is researching the possibilities of an artificially conscious computer?

First question I should address probably is how does QuEST (QUalia Exploitation of Sensing Technology) relate to big data. One of the modern breakthroughs we’ve been investigating in the QuEST effort is deep learning. Deep learning can be thought of as an artificial neural network big data solution. Deep learning systems that have hundreds of millions of adjustable parameters learn by being exposed to millions of training examples (big data). The recent competitions in labeling images off the internet, speech processing or language translations (without programming in any grammatical rules) are all being won by deep learning solutions. The most common solutions in deep learning use supervised learning to find correlations of data values to label categories that are pre-determined. For example, in a recent competition millions of images were used to train deep neural networks to determine membership of an image in one of 1000 categories. Google, Facebook, Microsoft and Baidu all have extremely large deep learning efforts ongoing.

We also have been investigating deep neural networks to fuse and label military sensor data. But at their core deep learning solutions face all the challenges mentioned above that all big data solutions face. They don’t generate meaning – or at least not in a manner completely understandable by the human decision maker.

They only generate results that we’ve programed them to distinguish by correlation of data to categories (for example the thousand image net categories). The great advantage of big data / deep learning is that I let the data analysis determine those characteristics observable in the data that can distinguish between the categories that I need to know for a given decision.

As an example, in my prior life working in the machine detection of breast cancer we programmed in image characteristics of tissue that distinguished cancer versus normal (what is the texture of this tissue, how big is this group of dense pixels, …). In a big data/deep learning approach I would gather so much data (so many mammograms) that I feed the raw pixels into the system and it determines what image characteristics are pertinent to the cancer or no-cancer decision. That is enormously valuable and far more accurate.

The concern I expressed above with respect to deep learning can be related to recent publications on deep learning associated with what is called ‘adversarial examples’. What is the meaning of an image of a school bus to these networks? An adversarial example of what appears to be an image of a school bus to a human is called an ostrich by the network with very high confidence. So again I

must emphasize big data does NOT generate meaning in a manner that we can blindly use.

With respect to timing for QuEST: QuEST is an ongoing open discussion that ask the questions associated with why has machine learning not solved the problems we all hoped it would. We are looking for the engineering characteristics of consciousness that could be the key ingredient that leads to that human robustness in making meaning? In the end we envision QuEST agents that combine experiential based decision making (intuition – possibly based on deep learning/big data approaches) along with a representation that replicates the engineering characteristics of our conscious representation (imagined/simulated and situated representation).

QuEST is not a project/program – we don’t have a timeline to accomplish – we are working theoretical issues on a Theory of consciousness and we are also working on what are the engineering characteristics of consciousness that may be the key to human robustness? We are implementing solutions to driver problems in cyber security and structural health monitoring and sensor processing to test where there is an engineering advantage to the QuEST approach. So far QuEST has led to increased performance in areas where we have tested it (for example in malware detection or in structural health monitoring). Just as humans have a subconscious representation we seek QuEST agents that combine experiential based representations (possibly instantiations of big data/deep learning) with a representation that can account for the unexpected stimuli (possibly via these artificially conscious simulated/situated representations).

One example from above of a topic from QuEST is what is meaning? QuEST definition of meaning:

The meaning of a stimulus is the agent specific representational changes evoked by that stimulus in that agent. The update to the representation, evoked by the data, is the meaning of the stimulus to this agent. Meaning is NOT just the posting into the representation of the data it is all the resulting changes to the representation. For example, the evoking of tacit knowledge or a modification of the ongoing simulation (consciousness) is included in the meaning of a stimulus to an agent. [26, 30] Meaning is not static and changes over time. The meaning of a stimulus is different for a given agent depending on when it is presented to the agent.

One of our current QuEST efforts is called AC3 (AFRL Conscious Content Curation). It combines deep learning and QuEST Theory of Consciousness tenets to generate semantic meta-data for the purpose of content curation. It is our intent as the AC3 effort matures to co-locate with commercial researchers working content curation for short duration high intensity incubation efforts to expose our researchers to the scale and agility of the commercial world (commercial innovation) while exposing our commercial partners to the QuEST approach that they might find of commercial interest.

Unasked question but a concern on the premise: I’ll be covering how big data and analytics are improving situational awareness, target acquisition, etc

I don’t believe that maximal situation awareness is what we should be after Using the Mica Endsley idea of the levels of SA are perception (what is out there), comprehension (how is the stuff out there related to each other) and projection (what is the stuff out there going to do next) – I define awareness as the mutual information between reality and the internal representation of an agent (computer or human – so that mutual information could be with respect to IDs of what is out there or how what is out there is related to each other and/or what is out there going to do next) – any attempt to be completely aware will fail (sensor errors / obfuscation / natural variation in the measurements …)

What consciousness does for critters in nature is provide an imagined representation where much of its contents are inferred versus measured (and this is used for perception of what is out there as well as recalling what we might have experience before or imagining what might happen next) – so we are seeking in QuEST situation consciousness versus situation awareness – a stable consistent and useful representation versus one that is maximally aware much of what we consciously experience is inferred not measured in our sensors – we attempt in QuEST to generate such representations and believe they will provide the robustness – the human meaning we associate with the stimuli we observe when combined with the intuition based (subconscious) processing – in our QuEST efforts we consider big-data a spectacular implementation of ‘autistic’ capturing of prior experiences and a determination of what axes account for measurement variations – those are then used as the axes for a simulation that is an instantiation of an artificial conscious representation that is a simulation versus a posting of sensory measurements and it is situated

So in conclusion we see big data (to include deep learning) as a necessary but not a sufficient condition to achieve the autonomy we seek in the QuEST effort.

news summary (29)

Categories: Uncategorized

Weekly QuEST Discussion Topics and News, 9 Oct

October 8, 2015 Leave a comment

QuEST 9 Oct 2015

I want to have a pick up where we left off with a discussion on the article Translating Videos to Natural Language using Deep Recurrent NNs: Venugopalan / Xu /Donahue / RohrBack/ Mooney/ Saenko:

  • Solving the visual symbol grounding problem has long been a goal of artificial intelligence.
  • The field appears to be advancing closer to this goal with recent breakthroughs in deep learning for natural language grounding in static images.
  • In this paper, we propose to translate videos directly to sentences using a unified deep neural network with both convolutional and recurrent structure.
  • Described video datasets are scarce, and most existing methods have been applied to toy domains with a small vocabulary of possible words.
  • By transferring knowledge from 1.2M+ images with category labels and 100,000+ images with captions, our method is able to create sentence descriptions of open-domain videos with large vocabularies.
  • We compare our approach with recent work using language generation metrics, subject, verb, and object prediction accuracy, and a human evaluation.

As we hit this article we want to map to a discussion on the Types of Qualia / Computing with Qualiaimplications.  Specifically can we use the ‘thought vectors’ as a representation of Qualia-space ~ Q-space.  To have that discussion we need to review what we consider as types of Qualia and what sort of computations we hope to accomplish with that representation.

If there is time I might also bring in the article on DRAW from Google researchers -This paper introduces the Deep Recurrent Attentive Writer (DRAW) neural network architecture for image generation. DRAW networks combine a novel spatial attention mechanism that mimics the foveation of the human eye, with a sequential variational auto-encoding framework that allows for the iterative construction of complex images.  The system substantially improves on the state of the art for generative models on MNIST, and, when trained on the Street View House Numbers dataset, it generates images that cannot be distinguished from real data with the naked eye.

I’ve recently during these deliberations been reviewing our article on the life and death of ATR and hope for resurrection – to see how our thoughts have changed since 2008.  So we might take a moment to revisit those ideas.

news summary (28)

Categories: Uncategorized

Get every new post delivered to your Inbox.