Large-Scale Biologically Inspired VLSI Systems

In recent years there is a growing interest for neuroscience and the prospect of understanding the human brain, as well as for using these findings to address some limitations of today’s technology. Principles of neural computation, which are defined by massive parallel processing, distributed information storage and adaptation can be utilized in order to enhance, or even revolutionize computer systems and technology. Realizing this great potentiality, especially the importance of understanding the human brain, the research community has lunched remarkable projects to support the field of computational neuroscience that studies the information processing properties of the nervous system. A notable ongoing project is the Blue Brain [1] at the Ecole Polytechnique Federale de Lausanne (EPFL), which seeks to simulate 10.000 neurons of the rat brain, by using detailed studies of the nervous system. Another project at IBM Almaden Research Center in California, focused on understanding the cortex that is the outer processing layer of the brain. Biological neuron models are employed for the simulation of 900 million neurons connected by 9 trillion synapses [2]. Both projects lead to great research findings, but rely on large-scale simulations in High-Performance Computing (HPC) clusters [3], known as supercomputers, like the IBM BlueGene/P. These simulations have the disadvantage of requiring a lot of execution time, since the simulation of seconds brain activity requires many minutes of processing on these computers. This is a result of the complexity given by the large amount of studied parameters of the neural model and the drawbacks of the von-Neumann computing architecture that all the HPC clusters employ. Taking into account the necessity of scaling up for the employed neural models to describe in more detail the brain function and structure, this drawback could be critical calling for alternative approaches. Moreover the energy harvesting computation of HPC clusters is another important issue that increases significantly the cost of these simulations.  With the existing HPC technology the simulation of a neural model that describes the activity in the scale of a whole brain, would require a supercomputer 1.000 times more powerful as the best existing today, with power requirements equal to the energy that a large city needs [4].

 

Neuromorphic systems use a radical different architecture compared to the conventional computer systems. Their structure and function resemble the nervous system, thereby advancing enormously the realism to study neural models.  This allows to move from simulations of a neural model to emulations, by using brain-like hardware systems that duplicate more faithfully the behavior of neural networks. Neuromorphic systems can mimic more closely the parallel processing of biological neural networks. Moreover their re-configurability is an important asset to replicate the plasticity and flexibility that the human brain demonstrates. The most important difference between the neuromorphic and conventional systems is that the latter use the von-Neumann architecture with the central processing unit to be physically separated from the memory, whereas the former are characterized by a distributing memory; the synapses of the network can implement at the same time memory and complex computation [5]. The rapid development of very-large-scale integration technology (VLSI) for chip fabrication has given the opportunity to place hundreds of thousands of electronic components on a single circuitry. This increases significantly the density of computational components and it can boost the size of neuromorphic systems that are steadily increased to handle progressively more advanced computational tasks. Dedicated hardware can offer a high-speed execution of large neural models in silicon in an affordable way compared to HPC clusters. All the above allow the large-scale neuromorphic systems to represent an attractive alternative to conventional numerical software simulations (HPC), which face the problems of increasing computational times, power consumption and model scalability.

 

Research publications in this field have greatly increased in number, which implies the strong potential of neuromorphic engineering. One research project that aims to develop a neuromorphic system is the  Neurogrid [6] from Stanford University. The project exploits the non-linear characteristics of the transistors in order to emulate the behavior of real neural cells in silicon. The final hardware platform will be a multi-chip neuromorphic system that can implement and emulate many different cortical areas in real time and reveal their interactions. This approach uses sub-threshold mixed-signal circuits that consume the same amount of current as the cells in biology. This approach slows down the execution speed to a biological realistic rate, providing the ground for the simulation of million cortical neurons in real-time. An asynchronous-multicast digital communication implements the synaptic connections of the network. A variety of neuronal cells and synaptic interactions can be implemented taking advantage from the full re-programmability of the system. However external resources must be used for the realization of synaptic plasticity mechanisms, since the system employs currently only linear synapses [5]. The aim of the Neurogrid project is the simulation in real-time of millions of neurons connected by billions of synapses in an affordable way, providing the benefit of a power efficient system with energy consumption orders of magnitude less than in HPC clusters.

 

Another notable project run from University of Manchester seeks to address the same problem by using commodity digital microprocessors connected to a dedicated pulse communication architecture for neural simulations. The project is called  SpiNNaker [7] and it aims to model brain activity in real-time, offering high flexibility and programmability like a general purpose computer does. For this purpose a computing platform is being developed that offers 57.600 custom-designed chips that can implement 18 low-power ARM9 processor cores. At the center of each chip a dedicated router receives and forwards all the incoming packets from/to the neighboring counterparts. A synchronous dynamic RAM on the top of each chip holds the connectivity information for up to 16 million synaptic connections. This digital system approach has mainly two advantages. First is the optimized routing and distribution of data packets (or events) with a dedicated global asynchronous communication [8]. Second is the high programmability of the overall digital system that is particularly viable for the replication of natural flexibility and plasticity of the brain. However this system approach due its conventional architecture, reflects the boundaries of a typical von Neumann computing system. The data transfer rate (throughput) between the processor and the memory is limited due to the shared memory bus. In particular for this system the more complex the simulated model is, the fewer elements can be simulated by the given platform (limited scaling).

 

An alternative to the above approach would be a neuromorphic system build from highly integrated mixed-signal circuits for the emulation of individual neurons, using a dedicated communication for the inter-connectivity. Such a system is being developed from the University of Heidelberg (UHEI) and Technische Universität Dresden (TUD), consisting the hardware platform of the BrainScaleS project. This approach uses wafer-scale integration technology [9] and custom-made analog circuits to replicate the behavior of neural cells at large-scale with multiple silicon interconnected wafers. Each wafer module implements up to 200.000 analog neurons and 40 million learning synapses, interconnected via a high-density routing grid on wafer. An important feature of this system is the accelerated computation compared to biological real time by a factor of 10.000. This way the required simulation time of an implemented neural network is shrinked from hours to seconds. The proposed system allows for power efficiency taking advantage from the analog neural computation of dedicated implemented hardware comparing to HPC clusters. This along with the provided sufficient re-configurability and the accelerated computation forms an attractive alternative over the simulations in HPC clusters. However the system requires large bandwidth achievements and fast digital circuits for the communication in order to cope with the performed acceleration. The overall system represents a flexible research platform to study the dynamics of large-scale biologically inspired neural networks with an affordable way.

 

Finally another notable system developed for the simulation of large-scale neural networks in real-time is the TrueNorth from IBM [10]. The project takes inspiration from the structure of biological neural systems, departing this way from a typical von Neumann architecture, in order to deliver a new hardware architecture and computational paradigm. One single chip consists of 4.096 cores, with each to simulate up to 256 neurons and implement 256×256 programmable synaptic connections. The chip is implemented fully digital and the routing of the events is asynchronous, but not flexible in the same degree as in SpiNNaker. The overall system composed from a multi-chip infrastructure demonstrates a massive parallel computation with fault tolerance and distributed memory. However, the so far implemented synapses do not realize any plasticity mechanisms making any online learning impossible [5].

 

From all the described above systems are derived four different kind of approaches for developing large-scale neuromorphic systems; custom mixed-signal [9], custom subthershold analog [6], custom fully digital [10] and conventional microprocessors [7]. The mixed-signal can refer to analog neural computation with digital implemented synapses [9], [6]. It can provide in same extend physical implementations of neural models and thus a platform for neural emulation. A fully digital approach implements both neurons and synapses as digital circuitry, offering a binary implementation of the model and simulation of the neural model. Design parameters vary from approach to approach. A custom digital system is much more re-configurable than a mixed-signal or an analog one. On the other hand the analog or mixed-signal systems are more power efficient than a digital one. However, as the CMOS technology advances towards better technology (referred to CMOS semiconductor device fabrication), this power gap between analog and digital domain is minimized. Furthermore, as concerns the integration density of components, it is more or less for all the above approaches at the same level. Except from the differences all the approaches share some common features like the massive parallelism, the configurability and the asynchronous communication.

 

[1] “The Blue Brain Project.” [Online]. Available: http://bluebrain.epfl.ch/

[2] R. Ananthanarayanan, S. K. Esser, H. D. Simon, and D. S. Modha, “The cat is out of the bag: cortical simulations with 109 neurons, 1013 synapses,” in Proceedings of the Conference on High Performance Computing Networking, Storage and Analysis, pp. 1-12, Nov. 2009.

[3] Pablo García-Risueño, Pablo E. Ibáñez, “A review of high performance computing foundations for scientists,” International Journal of Modern Physics C, vol. 23, no. 07, p. 1230001, July 2012.

[4] S. Furber, “To build a brain,” IEEE Spectrum, 2012.

[5] G. Indiveri and S.-C. Liu, “Memory and information processing in neuromorphic systems.”, Proceedings of the IEEE, vol. 103, no. 8, pp. 1379-1397, Aug. 2015.

[6] B. Benjamin, P. Gao, E. McQuinn, S. Choudhary, A. Chandrasekaran, J.-M. Bussat, R. Alvarez-Icaza, J. Arthur, P. Merolla, and K. Boahen, “Neurogrid: A mixed-analog digital multichip system for large-scale neural simulations,” Proceedings of the IEEE, vol. 102, no. 5, pp. 699–716, May 2014.

[7] S. Furber, F. Galluppi, S. Temple, and L. Plana, “The SpiNNaker Project,” Proceedings of the IEEE, vol. 102, no. 5, pp. 652–665, May 2014.

[8] L. Plana, S. B. Furber, S. Temple, M. Khan, Y. Shi, J. Wu, S. Yang, et al., “A GALS infrastructure for a massively parallel multiprocessor,” Design & Test of Computers, IEEE, vol. 24, no. 5, pp. 454-463, 2007.

[9] J. Schemmel and D. Briiderle and A. Griibl and M. Hock and K. Meier and S. Millner, “A wafer-scale neuromorphic hardware system for large-scale neural modeling,” in IEEE International Symposium on Circuits and Systems (ISCAS’10), pp. 1947–1950, 2010.

[10] P. Merolla, J. Arthur, R. Alvarez-Icaza, A. Cassidy, J. Sawada, F. Akopyan, B. Jackson, N. Imam, C. Guo, Y. Nakamura, B. Brezzo, I. Vo, S. Esser, R. Appuswamy, B. Taba, A. Amir, M. Flickner, W. Risk, R. Manohar, and D. Modha, “A million spiking-neuron integrated circuit with a scalable communication network and interface,” Science, pp. 668–673, August 2014.

 

Neuromorphic weekly: Neurala demo day and neuromorphic bug eye cam

This week, two big news for neuromorphic engineering ! The first one is about a company that I have been following for a while, Neurala. They took part to the Tech Stars run this year in Boston (it’s an incubator for startups like Y Combinator) and they recently made a presentation at their demo day. I will soon publish a more in-depth article about their technology, but for now here is the video of their presentation:

The other news was this article about the neuromorphic bug eye cam. This “neuromorphic” sensor can provide a 180 degrees vision to drones for example. The following video gives an overview of the features of this sensor, and shows some examples where it is used on a robot:

Neuromorphic Weekly: News about Qualcomm, memristors, and SpiNNaker in action

Not a lot of news this week, but there are some interesting links I found on neuromorphic engineering.

News

We know that Qualcomm is working on neuromorphic chips, via a company called the Brain Corporation. They recently said in an article that their work is related to the BRAIN initiative, and that they believe telecoms and neuroscience actually intersect. They also say that they want to “create smart consumer products powered by these artificial nervous systems”. So as always, they say they are working on neuromorphic engineering without actually saying much. But I am excited to hear more about their projects in the future.

Papers

I found an interesting paper on how memristors can be used to emulate STDP in neuromorphic systems: STDP and STDP variations with memristors for spiking neuromorphic learning systems

Conferences

The 2013 edition of the Capo Caccia Neuromorphic Engineering workshop is already over ! However, I found some interesting pages that are freely accessible online:

– Some real-world results from using the SpiNNaker robot, for example interfacing it with a small mobile robot: SpiNNaker at Capo Caccia. They even made a video about the project:

– Another workgroup focused on how to interface a drone with neuromorphic hardware

That’s it for this week, if you know about other exciting news about neuromorphic engineering please share !

Where to learn about neuromorphic engineering ?

I personally first heard about neuromorphic engineering four years ago, just before starting my PhD on the subject. I remember that it was difficult to find some informations about it, as it was only a field that some universities around the world were working on. But most of all, it was nearly impossible to find a good course to learn the basics of the field. Luckily, the situation has changed today, and there are some good courses and ressources out there to get you started, or to extend your knowledge on the field.

The course, nearly the only course that is really specific to neuromorphic engineering is the course published online by the ETHZ in Zürich. The course is updated every semester, and will really guide you from start to finish in neuromorphic engineering: it goes from basic knowledge of CMOS technology to the design of silicon neurons. You can find the course at the following address: Neuromorphic Engineering courses.

Another online ressource that I really like when it comes to neuromorphic engineering is the Frontiers of Neuromorphic Engineering website. As for all the Frontiers journals, all the papers are accessible by everybody. There are new papers coming in regularly from many research groups around the world, with a very good quality of papers overall. You can just dive in the different papers to learn so much about neuromorphic engineering. You can find it at the following address: Frontiers of Neuromorphic Engineering.

Finally, another way to learn more about neuromorphic engineering is simply to learn about what the fundamental components of the field: neuroscience, computer science, and electronics. For computer science, I recommend the following course on Coursera. For the two other fields, there are nice beginners courses on edX about computer science and electronics.

Do you know any other way to learn more about neuromorphic engineering ? Then leave a comment !

 

Neuromorphic Weekly: IBM neuromorphic chips, Human Brain Project & the BRAIN initiative

Welcome to this second edition of Neuromorphic Weekly ! This post is published every week, and is all about the latest news and published papers in the world of neuromorphic engineering. In this edition, some news about the Human Brain Project, a nice selection of neuromorphic-related papers, and the last week of the CNE workshop.

News

The Canadian website itbusiness published an article about the neuromorphic chips developed by IBM. Nothing really new, but it is a good article if you never heard about these chips (They are built using IBM’s 45 nm technology, and are 100 % digital).

The New York Times published an article about the Human Brain Project, called Bringing a virtual brain to life. The author also discusses a little bit about the BRAIN initiative in the US.

The blog Neurdon published an article about why money matters for robots.

Papers

Nothing new this week, but here are a selection of papers I particularly liked recently:

Learn about what neural networks could be implemented on neuromorphic chips in Six Networks on an Universal Neuromorphic Computing Substrate.

If you want to know more about the routing problem on a neuromorphic core, you can read the following paper: Population-based routing in the SpiNNaker neuromorphic architecture.

Events, conferences & workshops

It is the second week of the CapoCaccia Cognitive Neuromorphic Engineering Workshop ! It’s not possible to join anymore, but for  having been a participant for many years, I can tell you it’s a very interesting workshop and I encourage you to participate next year. Also, some of the outcome of the discussion groups are available online and are updated in live during the workshop, so you can check that out if you’re interested.

That’s it for this edition of Neuromorphic Weekly, hope you enjoyed it. If you have any relevant information or paper that you think should absolutely be in this weekly article, don’t hesitate to contact us directly !

Neuromorphic Weekly: First Edition

This is the very first edition of Neuromorphic Weekly, which will consist in a post published every week, where we speak about the latest news and published papers in the world of neuromorphic engineering. I truly believe that doing so will help the neuromorphic engineering community to know more about what’s going on in the different research groups around the world & in the industry. I haven’t published a lot on the blog during the last weeks, so I will speak about some news that occurred a bit earlier this year.

News

I think one of the biggest news for the neuromorphic is the announcement of two major projects relative to Brain science: the Human Brain Project in Europe (which I already mentioned in a previous article), and the BRAIN initiative in the US. Both will help a lot to make new advances in the field of neuromorphic engineering. For example, Qualcomm recently joined the BRAIN initiative, and it is well known that Qualcomm is working on neuromorphic engineering projects.

Still in the US, the Neurogrid project is making some progress as well. The Neurogrid platform offers to emulate 1 million neurons in real-time, while only consuming 5W of power, which is nothing compared the power required to simulate the same amount of neurons on a supercomputer.

Finally, I wanted to speak about the Parallela platform, which I personally backed on Kickstarter. The Parallela platform is basically a board running Ubuntu on an ARM processor, but which also embeds a 16 or 64 cores processor that can be “easily” used. It is not a neuromorphic chip, but having a many-cores, low-power board that is open-source and can be used by everybody is a great opportunity to play with larger neural networks without having a supercomputer at home. For example, I can imagine great applications of this board for robots that are controlled by neural networks running on the board.

Papers

The Electronic Vision(s) team in Heidelberg and the LCN in Lausanne have a new paper on ArXiv.org, called Reward-based learning under hardware constraints – Using a RISC processor embedded in a neuromorphic substrate, where they speak about how to use an embedded processor in a neuromorphic chip to implement learning features. This approach gives the neuromorphic chip much more flexibility to implement learning rules, for example globally modulated STDP.

Another paper was recently put on ArXiv.org, called Integration of nanoscale memristor synapses in neuromorphic computing architectures. In this paper the authors speak about how we can use memresistors to design neuromorphic systems that are closer the real biological nervous systems.

The team at Georgia Tech also has a new article out, called Computing with networks of spiking neurons on a biophysically motivated floating-gate based neuromorphic integrated circuit. In this article they show once again that neuromorphic implementation of neural networks are much more power efficient than the digital simulation of the same network.

Events, conferences & workshops

The yearly CapoCaccia Cognitive Neuromorphic Engineering Workshop is currently running. It’s not possible to join anymore, but for  having been a participant for many years, I can tell you it’s a very interesting workshop and I encourage you to participate next year. Also, some of the outcome of the discussion groups are available online and are updated in live during the workshop, so you can check that out if you’re interested.

That’s it for this first edition of Neuromorphic Weekly, hope you enjoyed it. If you have any relevant information or paper that you think should absolutely be in this weekly article, don’t hesitate to contact us directly !

What the Human Brain Project means for neuromorphic engineering

Yesterday was a great day for the field of neuromorphic engineering: the Human Brain Project, which will receive 1 billion euros over 10 years, was officially funded by the European Union.

The project is not only about neuromorphic engineering, but a large part of the funding will go directly into research related to neuromorphic systems. According to the website of the project, three areas will include research on neuromorphic systems.

The first one is neuromorphic computing systems, which is really close to recent European projects like the FACETS or the BrainScaleS projects. This part of the Human Brain Project aims to develop new computing systems, that can learn new tasks by themselves and process data more efficiently. For example, this could be applied to computer vision, data mining or real-time analysis of financial data. This will also allow the creation of low-power intelligent systems, which can be used in cars or for industrial robots.

Read more

Industrial interest in Neuromorphic Engineering

I had the idea of writing this article while talking to a friend over lunch. We were talking about which position I could take in industry, as a PhD student that worked for 3 years in the field of neuromorphic engineering. So naturally, he asked me : are there any big companies working in my field ? and if yes, what are the big players in the field of neuromorphic engineering ? Of course, I knew the answer, but when I got home I decided to look deeper into the subject, to find out who exactly is working in the field in industry and what exactly they are doing.

The first company I thought about, and the one I talked about with my friend, is of course IBM. They are clearly working on developing neuromorphic chips : we can just look at their recent publications. One of their most recent recent publication on the subject is called A Digital Neurosynaptic Core Using Event-Driven QDI Circuits. In this paper, they describe their most recent neuromorphic chip, which is built on a modular component called “neurosynaptic core” and is entirely digital, with up to 256 neurons and 1024*256 synapses. They also demonstrate the capabilities of the chip with a sound localization application. For me, IBM is one of the main challenger in the neuromorphic field, as they have immense resources, by accessing also in-house fabs! It is clear that IBM has the will to push the field further.

Read more

Future applications of neuromorphic engineering

Recently I have been asking myself : what could be the higher-level applications of neuromorphic engineering ? Indeed, research groups around the world are now developing larger and larger neuromorphic systems, but the applications are still low-level : pattern recognition, data classification, simple robot navigation … But what are the applications of hardware neural networks that would really make an impact ?

The first one I could think about is digital virtual assistants. Having a virtual assistant is a big trend in the online business world. Basically, it is a physical person, that usually you never met, that will perform some simple tasks for you that does not require any physical action. These tasks are for example doing extensive research on a given topic, booking train tickets, or posting on your blog. And that’s exactly the functions a good neuromorphic system could emulate : it does not require human-like intelligence, but it is much more complex that what is currently done with neuromorphic systems. I see it as an app that you could have on your phone, which send your requests to a remote neuromorphic system, a bit like Siri for iOS. We can imagine specialized assistants on the same principle : a virtual doctor, a virtual lawyer …

Read more

What is neuromorphic engineering ?

Today, we wanted to post about an essential question : what is neuromorphic engineering ? Here is a more detailed answer. In recent years we have witnessed efforts from the engineering community to build hardware systems, capable for large-scale neural modelling. This has resulted to the emergence of new class VLSI systems, the so-called neuromorphic systems. These systems perform artificial computation based on the principles of biology. Neuromorphic engineering takes inspiration from biology, physics, mathematics, computer science and electrical engineering to give answers about the neurobiological circuits, new efficient computational paradigms, low power hardware solutions and new technologies with robustness to damage and learning capabilities. The potentiality of this domain is not simply to improve the related conventional threads of science but also to revolutionize them.

Read more