System designed to label visual scenes turns out to detect particular objects too

 

The first layers (1 and 2) of a neural network trained to classify scenes seem to be tuned to geometric patterns of increasing complexity, but the higher layers (3 and 4) appear to be picking out particular classes of objects. Credit: Courtesy of the researchers Object recognition -- determining what objects are where in a digital image -- is a central research topic in computer vision. But a person looking at an image will spontaneously make a higher-level judgment about the scene as whole: It's a kitchen, or a campsite, or a conference room. Among computer science researchers, the problem known as "scene recognition" has received relatively little attention. Last December, at the Annual Conference on Neural Information Processing Systems, MIT researchers announced the compilation of the world's largest database of images labeled according to scene type, with 7 million entries. By exploiting a machine-learning technique known as "deep learning" -- which is a revival of the classic artificial-intelligence technique of neural networks -- they used it to train the most successful scene-classifier yet, which was between 25 and 33 percent more accurate than its best predecessor. At the International Conference on Learning Representations this weekend, the researchers will present a new paper demonstrating that, en route to learning how to recognize scenes, their system also learned how to recognize objects. The work implies that at the very least, scene-recognition and object-recognition systems could work in concert. But it also holds out the possibility that they could prove to be mutually reinforcing. "Deep learning works very well, but it's very hard to understand why it works -- what is the internal representation that the network is building," says Antonio Torralba, an associate professor of computer science and engineering at MIT and a senior author on the new paper. "It could be that the representations for scenes are parts of scenes that don't make any sense, like corners or pieces of objects. But it could be that it's objects: To know that something is a bedroom, you need to see the bed; to know that something is a conference room, you need to see a table and chairs. That's what we found, that the network is really finding these objects." Torralba is joined on the new paper by first author Bolei Zhou, a graduate student in electrical engineering and computer science; Aude Oliva, a principal research scientist, and Agata Lapedriza, a visiting scientist, both at MIT's Computer Science and Artificial Intelligence Laboratory; and Aditya Khosla, another graduate student in Torralba's group. Under the hood Like all machine-learning systems, neural networks try to identify features of training data that correlate with annotations performed by human beings -- transcriptions of voice recordings, for instance, or scene or object labels associated with images. But unlike the machine-learning systems that produced, say, the voice-recognition software common in today's cellphones, neural nets make no prior assumptions about what those features will look like. That sounds like a recipe for disaster, as the system could end up churning away on irrelevant features in a vain hunt for correlations. But instead of deriving a sense of direction from human guidance, neural networks derive it from their structure. They're organized into layers: Banks of processing units -- loosely modeled on neurons in the brain -- in each layer perform random computations on the data they're fed. But they then feed their results to the next layer, and so on, until the outputs of the final layer are measured against the data annotations. As the network receives more data, it readjusts its internal settings to try to produce more accurate predictions. After the MIT researchers' network had processed millions of input images, readjusting its internal settings all the while, it was about 50 percent accurate at labeling scenes -- where human beings are only 80 percent accurate, since they can disagree about high-level scene labels. But the researchers didn't know how their network was doing what it was doing. The units in a neural network, however, respond differentially to different inputs. If a unit is tuned to a particular visual feature, it won't respond at all if the feature is entirely absent from a particular input. If the feature is clearly present, it will respond forcefully. The MIT researchers identified the 60 images that produced the strongest response in each unit of their network; then, to avoid biasing, they sent the collections of images to paid workers on Amazon's Mechanical Turk crowdsourcing site, who they asked to identify commonalities among the images. Beyond category "The first layer, more than half of the units are tuned to simple elements -- lines, or simple colors," Torralba says. "As you move up in the network, you start finding more and more objects. And there are other things, like regions or surfaces, that could be things like grass or clothes. So they're still highly semantic, and you also see an increase." According to the assessments by the Mechanical Turk workers, about half of the units at the top of the network are tuned to particular objects. "The other half, either they detect objects but don't do it very well, or we just don't know what they are doing," Torralba says. "They may be detecting pieces that we don't know how to name. Or it may be that the network hasn't fully converged, fully learned." In ongoing work, the researchers are starting from scratch and retraining their network on the same data sets, to see if it consistently converges on the same objects, or whether it can randomly evolve in different directions that still produce good predictions. They're also exploring whether object detection and scene detection can feed back into each other, to improve the performance of both. "But we want to do that in a way that doesn't force the network to do something that it doesn't want to do," Torralba says. Story Source: The above story is based on materials provided by Massachusetts Institute of Technology. The original article was written by Larry Hardesty. Note: Materials may be edited for content and length.

Smaller, faster, greener "high-rise" 3D chips are ready for Big Data

 

A four-story 3-D chip designed at Stanford could help address the current data processing limitations of today's technology (Image: Stanford University) Stanford engineers have pioneered a new design for a scalable 3D computer chip that tightly interconnects logic and memory, with the effect of minimizing data bottlenecks and saving on energy usage. With further work, the advance could be the key to a very substantial jump in performance, efficiency, and the ability to quickly process very large amounts of information  –  known as "Big Data"  –  over conventional chips. Bottlenecks A chain is only as strong as its weakest link. In the context of chip design, that link is usually the data bus that connects the memory and logic components and fetches data from memory, delivers it to the logic units for processing, and then sends it back over for storage. As it turns out, both CPU speeds and memory size are advancing at a much faster rate than the throughput speed of the data bus is improving. This means that when processing large amounts of data the CPU has to slow down to a crawl, constantly waiting for new data to arrive, wasting a lot of time and energy in the process. And things are only going to get worse as the gap increases. Couple this trend with the huge interest in Big Data in recent years and it’s easy to see how, if nothing is done to address this, we might end up with a serious problem on our hands. One way to deal with the data bottleneck could be to ferry data inside a chip using the much faster optical fibers, though the technology still seems quite far from mass production. Instead, a team led by professors Subhasish Mitra and Philip Wong at Stanford is exploring the more radical avenue of creating dense 3D chips that integrate memory and logic right on top of each other, and that exchange data using an array of thousands of vertical nanoscale interconnections. The sheer number of short-distance connections allow the data to travel much faster, using less electricity, and sidestepping the bottleneck problem almost entirely. A standard one-story chip versus an exploded view of Stanford's multi-story design (Image: Stanford University) "The memory is now right on top of the logic, so the data doesn’t have to move back and forth across long distances," lead author of the paper Max Shulaker tells Gizmag. "Putting them so close together (vertically over one another) therefore saves a lot of energy in passing data back and forth between the logic and memory. Also, now that the bandwidth between the memory and logic is greatly increased, the processor doesn’t have to waste time or energy waiting to get the data." A prototype 3D chip built by the researchers packs four layers on top of each other, two layers of memory "sandwiched" between two layers of logic. This shows that the researchers are able to stack all possible combinations on top of each other  –  memory on logic, memory on memory, logic on memory, and logic over logic. "But you could build even more layers using the exact same methodology depending on what your application required," says Shulaker. A matter of temperature This sort of close-quarters stacking is not possible in your standard chip because manufacturing a memory chip requires very high temperatures, on the order of 1,000° C (1,800° F), which would melt down the layer below it. So the team opted for a special type of memory which they had previously developed, called RRAM (for "resistive random access memory"). It doesn’t use silicon, but rather a combination of titanium nitride, hafnium oxide and platinum. Applying electricity to the memory cell one way causes it to resist the flow of electricity (hence the name), which equates to a "0" bit, while applying voltage the opposite way causes the structure to conduct again  –  representing a "1" bit. Apart from consuming less energy, the advantage of RRAM is that it can be built at much lower temperatures, meaning it can be manufactured right on top of other circuits, paving the way for building functional 3D chips. Interestingly, the transistors which are part of the "high-rise" design can be either standard silicon field-effect transistors (Si-FET) or made out of the much more energy-efficient carbon nanotubes (CNFET). In a previous study, Mitra, Wong and colleagues found a way to manufacture what they say are some of the highest-performing nanotube-based transistors to date. In the CNTFETs, multiple parallel carbon nanotubes replace silicon as the transistor channel (Image: Stanford University) "We benchmarked our CNFETs against silicon-based transistors which are in production today," Shulaker tells us. "We take the foundry models (the models the companies that make the silicon transistors use) for the silicon transistors, and change parameters in their models to match the CNT transistors we make in our lab at Stanford (since industry labs can print much smaller transistors than we can in an academic lab), and we see that our CNT transistors are now competitive with these silicon transistors." Previous CNTFETs could not reach high levels of performance because the concentration of carbon nanotubes was too low to build an effective chip. The Stanford team used a simple but ingenious workaround: the researchers started growing the nanotubes as usual and then used a sort of metal "Scotch tape" to transfer the nanotubes onto a silicon wafer that would serve as the base of the chip. Repeating the process 13 times per wafer resulted in a very high-density carbon nanotube grid. "It is projected that CNT transistors will achieve an order of magnitude benefit in energy-delay product  –  a metric of energy efficiency  –  compared to silicon CMOS once we can work out the remaining obstacles," Shulaker continues. "Thus, for one third the amount of energy, your circuit would run three times faster, for instance." Interconnections As the scientists deposited each memory layer, they were also able to create thousands of nanoscale interconnections into the logic layer below, which are meant to serve the role of the data bus. The great number of connections, along with the extremely short distances that data has to travel, allows the chip to avoid the data "traffic jams" that are plaguing current chip designers. As you can see from the figure above, the interconnections between the logic and memory can be arbitrary. In the first layer from the bottom, the logic layer is made out of standard (Si-FET) transistors, demonstrating how this design can integrate with existing technology. Crunching Big Data This research is still in its early stages, but the scientists say their design and manufacturing techniques are scalable and could lead to a significant leap in computing performance. "Monolithic 3D integration of logic and memory and emerging nanotechnologies like CNT transistors are promising steps for building the next generation of ultra-high efficiency and high performance electronic systems that can operate on massive amounts of data," says Shulaker. "The ability to operate on massive amounts of data in an energy-efficient manner could enable new applications that we can’t dream of today." The next step for the team will be to use this integration scheme to demonstrate new systems that cannot be built using today’s technologies and which leverage the data-crunching abilities of such systems. "Paradigm shift is an overused concept, but here it is appropriate," says Prof. Wong. "With this new architecture, electronics manufacturers could put the power of a supercomputer in your hand." Two papers describing their advance were presented at the IEEE International Electron Devices Meeting (IEDM) on December 15–17. Source: Stanford University

New discovery could pave way for spin-based computing: Novel oxide-based magnetism follows electrical commands

 

---

Magnetic states at oxide interfaces controlled by electricity. Top image show magnetic state with -3 volts applied, and bottom image shows nonmagnetic state with 0 volts applied.

Electricity and magnetism rule our digital world. Semiconductors process electrical information, while magnetic materials enable long-term data storage. A University of Pittsburgh research team has discovered a way to fuse these two distinct properties in a single material, paving the way for new ultrahigh density storage and computing architectures.

While phones and laptops rely on electricity to process and temporarily store information, long-term data storage is still largely achieved via magnetism. Discs coated with magnetic material are locally oriented (e.g. North or South to represent "1" and "0"), and each independent magnet can be used to store a single bit of information. However, this information is not directly coupled to the semiconductors used to process information. Having a magnetic material that can store and process information would enable new forms of hybrid storage and processing capabilities.

Such a material has been created by the Pitt research team led by Jeremy Levy, a Distinguished Professor of Condensed Matter Physics in Pitt's Kenneth P. Dietrich School of Arts and Sciences and director of the Pittsburgh Quantum Institute.

Levy, other researchers at Pitt, and colleagues at the University of Wisconsin-Madison today published their work in Nature Communications, elucidating their discovery of a form of magnetism that can be stabilized with electric fields rather than magnetic fields. Working with a material formed from a thick layer of one oxide -- strontium titanate -- and a thin layer of a second material -- lanthanum aluminate -- these researchers have found that the interface between these materials can exhibit magnetic behavior that is stable at room temperature. The interface is normally conducting, but by "chasing" away the electrons with an applied voltage (equivalent to that of two AA batteries), the material becomes insulating and magnetic. The magnetic properties are detected using "magnetic force microscopy," an imaging technique that scans a tiny magnet over the material to gauge the relative attraction or repulsion from the magnetic layer.

The newly discovered magnetic properties come on the heels of a previous invention by Levy, so-called "Etch-a-Sketch Nanoelectronics" involving the same material. The discovery of magnetic properties can now be combined with ultra-small transistors, terahertz detectors, and single-electron devices previously demonstrated.

"This work is indeed very promising and may lead to a new type of magnetic storage," says Stuart Wolf, head of the nanoSTAR Institute at the University of Virginia. Though not an author on this paper, Wolf is widely regarded as a pioneer in the area of spintronics.

"Magnetic materials tend to respond to magnetic fields and are not so sensitive to electrical influences," Levy says. "What we have discovered is that a new family of oxide-based materials can completely change its behavior based on electrical input."

---

Story Source:

The above story is based on materials provided by University of Pittsburgh. Note: Materials may be edited for content and length.

---

Journal Reference:

1. Feng Bi, Mengchen Huang, Sangwoo Ryu, Hyungwoo Lee, Chung-Wung Bark, Chang-Beom Eom, Patrick Irvin, Jeremy Levy. Room-temperature electronically-controlled ferromagnetism at the LaAlO3/SrTiO3 interface. Nature Communications, 2014; 5: 5019 DOI: 10.1038/ncomms6019

---

 

Can our computers continue to get smaller and more powerful?

 

University of Michigan computer scientist reviews frontier technologies to determine fundamental limits of computer scaling

Algorithms help optimize the placement of parts on an integrated circuit--a way to continue scaling.

August 13, 2014

From their origins in the 1940s as sequestered, room-sized machines designed for military and scientific use, computers have made a rapid march into the mainstream, radically transforming industry, commerce, entertainment and governance while shrinking to become ubiquitous handheld portals to the world.

This progress has been driven by the industry's ability to continually innovate techniques for packing increasing amounts of computational circuitry into smaller and denser microchips. But with miniature computer processors now containing millions of closely-packed transistor components of near atomic size, chip designers are facing both engineering and fundamental limits that have become barriers to the continued improvement of computer performance.

Have we reached the limits to computation?

In a review article in this week's issue of the journal Nature, Igor Markov of the University of Michigan reviews limiting factors in the development of computing systems to help determine what is achievable, identifying "loose" limits and viable opportunities for advancements through the use of emerging technologies. His research for this project was funded in part by the National Science Foundation (NSF).

"Just as the second law of thermodynamics was inspired by the discovery of heat engines during the industrial revolution, we are poised to identify fundamental laws that could enunciate the limits of computation in the present information age," says Sankar Basu, a program director in NSF's Computer and Information Science and Engineering Directorate. "Markov's paper revolves around this important intellectual question of our time and briefly touches upon most threads of scientific work leading up to it."

The article summarizes and examines limitations in the areas of manufacturing and engineering, design and validation, power and heat, time and space, as well as information and computational complexity.​

"What are these limits, and are some of them negotiable? On which assumptions are they based? How can they be overcome?" asks Markov. "Given the wealth of knowledge about limits to computation and complicated relations between such limits, it is important to measure both dominant and emerging technologies against them."

Limits related to materials and manufacturing are immediately perceptible. In a material layer ten atoms thick, missing one atom due to imprecise manufacturing changes electrical parameters by ten percent or more. Shrinking designs of this scale further inevitably leads to quantum physics and associated limits.

Limits related to engineering are dependent upon design decisions, technical abilities and the ability to validate designs. While very real, these limits are difficult to quantify. However, once the premises of a limit are understood, obstacles to improvement can potentially be eliminated. One such breakthrough has been in writing software to automatically find, diagnose and fix bugs in hardware designs.

Limits related to power and energy have been studied for many years, but only recently have chip designers found ways to improve the energy consumption of processors by temporarily turning off parts of the chip. There are many other clever tricks for saving energy during computation. But moving forward, silicon chips will not maintain the pace of improvement without radical changes. Atomic physics suggests intriguing possibilities but these are far beyond modern engineering capabilities.

Limits relating to time and space can be felt in practice. The speed of light, while a very large number, limits how fast data can travel. Traveling through copper wires and silicon transistors, a signal can no longer traverse a chip in one clock cycle today. A formula limiting parallel computation in terms of device size, communication speed and the number of available dimensions has been known for more than 20 years, but only recently has it become important now that transistors are faster than interconnections. This is why alternatives to conventional wires are being developed, but in the meantime mathematical optimization can be used to reduce the length of wires by rearranging transistors and other components.

Several key limits related to information and computational complexity have been reached by modern computers. Some categories of computational tasks are conjectured to be so difficult to solve that no proposed technology, not even quantum computing, promises consistent advantage. But studying each task individually often helps reformulate it for more efficient computation.

When a specific limit is approached and obstructs progress, understanding the assumptions made is key to circumventing it. Chip scaling will continue for the next few years, but each step forward will meet serious obstacles, some too powerful to circumvent.

What about breakthrough technologies? New techniques and materials can be helpful in several ways and can potentially be "game changers" with respect to traditional limits. For example, carbon nanotube transistors provide greater drive strength and can potentially reduce delay, decrease energy consumption and shrink the footprint of an overall circuit. On the other hand, fundamental limits--sometimes not initially anticipated--tend to obstruct new and emerging technologies, so it is important to understand them before promising a new revolution in power, performance and other factors.

"Understanding these important limits," says Markov, "will help us to bet on the right new techniques and technologies."

-NSF-

The brain: Key to a better computer

 

"Today's computers are wonderful at bookkeeping and solving scientific problems often described by partial differential equations, but they're horrible at just using common sense, seeing new patterns, dealing with ambiguity and making smart decisions," said John Wagner, cognitive sciences manager at Sandia National Laboratories.

In contrast, the brain is "proof that you can have a formidable computer that never stops learning, operates on the power of a 20-watt light bulb and can last a hundred years," he said.

Although brain-inspired computing is in its infancy, Sandia has included it in a long-term research project whose goal is future computer systems. Neuro-inspired computing seeks to develop algorithms that would run on computers that function more like a brain than a conventional computer.

"We're evaluating what the benefits would be of a system like this and considering what types of devices and architectures would be needed to enable it," said microsystems researcher Murat Okandan.

Sandia's facilities and past research make the laboratories a natural for this work: its Microsystems & Engineering Science Applications (MESA) complex, a fabrication facility that can build massively interconnected computational elements; its computer architecture group and its long history of designing and building supercomputers; strong cognitive neurosciences research, with expertise in such areas as brain-inspired algorithms; and its decades of work on nationally important problems, Wagner said.

New technology often is spurred by a particular need. Early conventional computing grew from the need for neutron diffusion simulations and weather prediction. Today, big data problems and remote autonomous and semiautonomous systems need far more computational power and better energy efficiency.

Neuro-inspired computers would be ideal for robots, remote sensors

Neuro-inspired computers would be ideal for operating such systems as unmanned aerial vehicles, robots and remote sensors, and solving big data problems, such as those the cyber world faces and analyzing transactions whizzing around the world, "looking at what's going where and for what reason," Okandan said.

Such computers would be able to detect patterns and anomalies, sensing what fits and what doesn't. Perhaps the computer wouldn't find the entire answer, but could wade through enormous amounts of data to point a human analyst in the right direction, Okandan said.

"If you do conventional computing, you are doing exact computations and exact computations only. If you're looking at neurocomputation, you are looking at history, or memories in your sort of innate way of looking at them, then making predictions on what's going to happen next," he said. "That's a very different realm."

Modern computers are largely calculating machines with a central processing unit and memory that stores both a program and data. They take a command from the program and data from the memory to execute the command, one step at a time, no matter how fast they run. Parallel and multicore computers can do more than one thing at a time but still use the same basic approach and remain very far removed from the way the brain routinely handles multiple problems concurrently.

The architecture of neuro-inspired computers would be fundamentally different, uniting processing and storage in a network architecture "so the pieces that are processing the data are the same pieces that are storing the data, and the data will be processed with all nodes functioning concurrently," Wagner said. "It won't be a serial step-by-step process; it'll be this network processing everything all at the same time. So it will be very efficient and very quick."

Unlike today's computers, neuro-inspired computers would inherently use the critical notion of time. "The things that you represent are not just static shots, but they are preceded by something and there's usually something that comes after them," creating episodic memory that links what happens when. This requires massive interconnectivity and a unique way of encoding information in the activity of the system itself, Okandan said.

More neurosciences research opens more possibilities for brain-inspired computing

Each neuron in a neural structure can have connections coming in from about 10,000 neurons, which in turn can connect to 10,000 other neurons in a dynamic way. Conventional computer transistors, on the other hand, connect on average to four other transistors in a static pattern.

Computer design has drawn from neuroscience before, but an explosion in neuroscience research in recent years opens more possibilities. While it's far from a complete picture, Okandan said what's known offers "more guidance in terms of how neural systems might be representing data and processing information" and clues about replicating those tasks in a different structure to address problems impossible to solve on today's systems.

Brain-inspired computing isn't the same as artificial intelligence, although a broad definition of artificial intelligence could encompass it.

"Where I think brain-inspired computing can start differentiating itself is where it really truly tries to take inspiration from biosystems, which have evolved over generations to be incredibly good at what they do and very robust against a component failure. They are very energy efficient and very good at dealing with real-world situations. Our current computers are very energy inefficient, they are very failure-prone due to components failing and they can't make sense of complex data sets," Okandan said.

Computers today do required computations without any sense of what the data is -- it's just a representation chosen by a programmer.

"Whereas if you think about neuro-inspired computing systems, the structure itself will have an internal representation of the datastream that it's receiving and previous history that it's seen, so ideally it will be able to make predictions on what the future states of that datastream should be, and have a sense for what the information represents." Okandan said.

He estimates a project dedicated to brain-inspired computing will develop early examples of a new architecture in the first several years, but said higher levels of complexity could take decades, even with the many efforts around the world working toward the same goal.

"The ultimate question is, 'What are the physical things in the biological system that let you think and act, what's the core essence of intelligence and thought?' That might take just a bit longer," he said.

Improved computer simulations enable better calculation of interfacial tension

 

---

At coexistence, the crystal (red) and the fluid (blue) are separated by interfaces. The simulation box shown here contains 3,660 hard sphere particles. Using periodic boundary conditions and finite-size scaling (systematic variation of the box size), computer simulations allow high precision measurements of the interfacial tension.

Researchers from Mainz University identify novel mechanisms of logarithmic finite-size corrections relevant to the determination of interfacial tension.

Computer simulations play an increasingly important role in the description and development of new materials. Yet, despite major advances in computer technology, the simulations in statistical physics are typically restricted to systems of up to a few 100,000 particles, which is many times smaller than the actual material quantities used in typical experiments. Researchers therefore use so-called finite-size corrections in order to adjust the results obtained for comparatively small simulation systems to the macroscopic scale. A team of researchers from Johannes Gutenberg University Mainz (JGU) has now succeeded in better understanding how this technique works when it is used to assess interfacial tension, thus enabling more accurate predictions.

The interfacial tension is an important physical quantity of many phenomena, such as the nucleation of water droplets in the atmosphere, the crystallization of proteins from solutions, and the growth and stability of nanocrystals. It occurs at the interface between different phases of a material, i.e., on the transition between solid, liquid, and gaseous phases. However, the interfacial tension is difficult to measure experimentally, and reliable analytical theories about it are also lacking. Thus it is of particular importance to develop computer simulation techniques for this phenomenon.

Using an innovative simulation method, Fabian Schmitz, Dr. Peter Virnau, and Professor Kurt Binder of the Condensed Matter Theory group at JGU's Institute of Physics have now succeeded in gaining a better understanding of the nature of finite-size corrections in the determination of interfacial tension. This work, achieved only after several million CPU hours on the Mainz supercomputer MOGON, will in the future help researchers to analyze interfacial tension with the highest precision by means of simulations. The results were published in the journal Physical Review Letters.

High-performance computing becomes increasingly important at Johannes Gutenberg University Mainz. The planned new supercomputer MOGON II is expected to replace the current system in the first quarter of 2016. It is expected that MOGON II will be among the top 100 fastest high-performance computers worldwide.

---

Story Source:

The above story is based on materials provided by Universität Mainz. Note: Materials may be edited for content and length.

---

Journal Reference:

1. Fabian Schmitz, Peter Virnau, Kurt Binder. Determination of the Origin and Magnitude of Logarithmic Finite-Size Effects on Interfacial Tension: Role of Interfacial Fluctuations and Domain Breathing. Physical Review Letters, 2014; 112 (12) DOI: 10.1103/PhysRevLett.112.125701