Home > Uncategorized > Weekly QuEST Discussion Topics and News, 10 Feb

Weekly QuEST Discussion Topics and News, 10 Feb

QuEST 10 Feb 2017

Will probably not discuss it (really down in the weeds) but will post on the site an article / presentation we’ve been banging on this week on reinforcement learning.

JX Wang  arXiv:1611.05763v3 [cs.LG] 23 Jan 2017

The goal is to attack the task flexibility issue and the large onerous amount of data issue for RL:

  • In recent years deep reinforcement learning (RL) systems have attained superhuman performance in a number of challenging task domains. However, a major limitation of such applications is their demand for massive amounts of training data. A critical present objective is thus to develop deep RL methods that can adapt rapidly to new tasks. In the present work we introduce a novel approach to this challenge, which we refer to as deep meta-reinforcement learning. Previous work has shown thatrecurrent networks can support meta-learning in a fully supervised context.
  • We extend this approach to the RL setting. What emerges is a system that istrained using one RL algorithm, but whose recurrent dynamics implement a second, quite separate RL procedure. This second, learned RL algorithm can differ from the original one in arbitrary ways. Importantly, because it is learned, it is configured to exploit structure in the training domain. We unpack these points in a series of seven proof-of-concept experiments, each of which examines a key aspect of deep meta-RL. We consider prospects for extending and scaling up the approach, and also point out some potentially important implications for neuroscience.

We will start the open discussion this week by discussing topics that the group suggest should be included in an upcoming presentation by Cap – “Artificial Intelligence and Machine Learning – Where are we?  How did we get here?  Where do we need to go?” – Cap will present a strawman version to get comments on flow and content and opinions on the ‘big rocks’ that should be included.

Next we need to discuss representation – we’ve suggested this is the key to autonomy – to have agents with a representation or set of representations to facilitate deliberation / decision making for robustness and for a common framework when part of a human-machine joint cognitive solution.

Representation:  how the agent structures what it knows about the world – so for example its knowledge (what it uses to generate meaning of an observable) 

Reasoning:  how the agent can change its representation – the manipulation of the representation for example for the generation of meaning

Understanding:  application of relevant parts of the representation to complete a task – the meaning generated by an agent relevant to accomplishing a task

This is all part of our continuing discussion of the Kabrisky lecture – What is QueST? – can a better representation be the missing link for recommender systems – instead of the representation being generated by optimizing an objective function tied to MSE on classification – imagine the objective function tied to stability, consistency and usefulness – this may be an approach to lead to  systems systems ‘appreciate’ the information in the data or the context (meaning) of the human’s environment / thoughts and current focus – thus they become an approach to overcome the  ‘feed’ – the human is sucking on the firehose data feed – social media example – but people can’t seem to disconnect (they don’t have the will power to disconnect) – if we design a joint cognitive social media system focused on ‘mindfulness’ and thus a ‘context aware feed’ that provides some value –  how can QuEST agents facilitate the human getting into the ‘zone’ – the illusory apparent slowdown in time – we conjecture that the conscious perception of time is associated with the efficient ‘chunking’ of experiences – thus a QuEST ‘wingman’ agent that helps the human formulate, recognize and exploit chunks would provide the insights to better respond to what may seem without it to be an overwhelming set of stimuli and evoke the zone illusion – thus our comment – a conscious recommender system facilitates the human decision maker getting into the zone

As part of the discussion on representation cap will present information from a recent article:

Brain-Computer Interface-Based
Communication in the Completely Locked-In
Ujwal Chaudhary

Chaudhary U, Xia B, Silvoni S, Cohen LG,

Birbaumer N (2017) Brain±Computer Interface±

Based Communication in the Completely Locked-In

State. PLoS Biol 15(1): e1002593. doi:10.1371/


  • Despite partial success, communication has remained impossible for persons suffering from complete motor paralysis but intact cognitive and emotional processing, a state called complete locked-in state (CLIS).
  • Based on a motor learning theoretical context and on the failure of neuroelectric brain-computer interface (BCI) communication attempts in CLIS, we here report BCI communication using functional near-infrared spectroscopy (fNIRS) and an implicit attentional processing procedure.
  • Four patients suffering from advanced amyotrophic lateral sclerosis (ALS)Ðtwo of them in permanent CLIS and two entering the CLIS without reliable means ofcommunicationÐlearned to answer personal questions with known answers andopen questions all requiring a ªyesº or ªnoº thought using frontocentral oxygenation changes measured with fNIRS.
  • Three patients completed more than 46 sessions spread over several weeks, and one patient (patient W) completed 20 sessions.
  • Online fNIRS classification of personal questions with known answers and open questions using linear support vector machine (SVM) resulted in an above-chance-level correct response rate over 70%.
  • Electroencephalographic oscillations and electrooculographic signals did not exceed the chance-level threshold for correct communication despite occasional differences between the physiological signals representing a ªyesº or ªnoº response. ** EEG not work **
  • However, electroencephalogram (EEG) changes in the theta-frequency band correlated with inferior communication performance, probably because of decreased vigilance and attention. If replicated with ALS patients in CLIS, these positive results could indicate the first step towards abolition of complete locked-in states, at least for ALS





Categories: Uncategorized
  1. No comments yet.
  1. No trackbacks yet.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: