Gene Frantz
TI Principal Fellow and Business Development Manager, DSP

When following the progression of modern processor architectures, it is possible to see where we may have lost our way. The introduction of the first microprocessor allowed us to process very simple signals in real time. During this time, the array processor and mini-computer were considered the state-of-the-art tools for signal processing and premier choice in the research community. As researchers discovered and developed new signal processing concepts and algorithms, another community of technologists were already on the path to creating the first DSP device. Approximately 20 years after the invention of the transistor, the first commercially available DSPs appeared on the market and dramatically changed the future of the world.

Early work in DSP algorithm implementation focused on reducing the number of multiplications since multiplies were expansive and slow when implemented in hardware. The primary breakthrough of widespread DSP adoption was the addition of a specialized hardware multiplier to the microprocessor. This innovation changed the focus of digital signal processing from reducing the number of multiplies required by an algorithm to instead, optimizing the numbers of necessary multiplies and additions.

Another major facet of the fast evolving DSP architecture was the use Harvard architecture (two busses – one for program memory and one for data memory) rather than Von Neuman architecture (a single bus with program and data sharing the same memory space). To better suit the needs of mathematically intense processors, the two busses of the Harvard architecture were modified to support both program and data memory allowing both busses to feed the multiplier.

To take advantage of the accelerated multiplication capabilities, new instructions were created to bring together the necessary operations to perform a multiply in a single instruction cycle. Later, the accumulate operation was added to create the familiar MAC function.

Further improvements to the DSP architecture followed. Combining the best of both the Harvard and Von Neuman architectures resulted in a multiple bus Von Neuman-style architecture. As IC performance increased, the concept of the Very Long Instruction Word (VLIW) was introduced allowing parallel processing to meet real-time constraints.

Modern architectures continue to press the limits of performance and efficiency through innovations such as deep pipelines, extensive branch prediction technology and advanced instruction sets. These innovations, in turn create new challenges even as they overcome previous limitations, which I plan to discuss in several future blogs.

Gene Frantz
TI Principal Fellow and Business Development Manager, DSP

Over the last several months, many of you have been asking if I were going to start blogging again. So here I am. Fortunately, I have a lot on my mind and look forward to re-engaging with you all. Over the last several months I’ve been traveling, including a recent trip to China. (Here is a picture of me while in China – I’m the one on the right.)

I’ve also been to Russia, Israel, Taiwan and other foreign places like Boston to visit MIT (Note that I am on a cruise in the Caribbean as I write this). But, this wasn’t my first year to extensively travel – I have traveled my whole career. One of the skills I have managed to develop is handling jetlag. I can now fly to anywhere in the world without jetlag. Actually this is not totally true. I notice that the first couple of days I am hungry at breakfast time. But after those couple of days I am back to only a cup of coffee (refilled many times) to be all I need. And, before you ask, no, it doesn’t matter which way I fly as I don’t have jetlag when I get home for the most part.

I have said all of this as I think there is a new phenomena happening to us today very similar to jetlag. That new phenomena is what I call “Net lag”. So, just as a term was created to describe the reaction of our bodies to the jet airplane, there needs to be a term to describe the problems that have been created by the Internet.

I am certain you have thought about some of them. For example, we have design teams in TI made up of people from Texas, California, Boston, Japan, India and Europe. Finding a time for weekly team meetings becomes somewhat difficult. Someone has to attend in the middle of their night while others are just awakening, and others are preparing for bed. The best I can tell no one seems to have the luck of the meeting being in the middle of their day.

Other examples

• sending and receiving email
• playing Internet games
• phone calls – we’ve gone from four digit phone numbers to seven to ten. Will we be able to remember phone numbers with more digits that we have on our hands?

This will only get worse with video phones and video conferencing. The idea of virtual teams will spread like wildfire. Employees will be able to live where they want while being an active part of the team.

What will be the casualty of this Net lag phenomena? Our health, our social life and our sense of community? But, of course, there will be great advantages.

So, I’ll stop here with this topic. But I invite you to send me other examples of Net lag and even good stories about it.

Gene Frantz
TI Principal Fellow and Business Development Manager, DSP

Smart, programmable cameras with a high degree of analytical intelligence will be the workhorses that enable the next generation of applications for the security and automotive markets to name just a couple.

Smart cameras will constantly be absorbing, processing and acting on information all around us, so we are free to think about other things. If someone climbs over a fence at 1:00 in the morning, these cameras will be able to distinguish between your teenage son sneaking home or a burglar trying to sneak off with your TV set. In an airport or other public places, they will know if a bag has been left unattended and will contact the appropriate authorities. On the streets, they will reduce the work load on the police force and other emergency services. In our cars they will serve as a security function but will also eventually take over the driving responsibilities. (Read my entry on, “Security vs. Safety and Privacy”)

Currently, when we think of video analytics, we think of a human as the end user. We think in terms of receiving information in a storable size that still contains all the data, of compression and reproduction. But the future of analytics will feature a computer as the end user. In that scenario, compression is no longer an issue and a perfect picture is unimportant – to a machine, a bad pixel is merely a spec of dust to be ignored. Analytics becomes less about data and more about intelligence, and that is a huge step that will require not just technological transition, but sociological transition as well. People may not be quite ready for machines to make important decisions without any human input. (Another good one to read, “Is Technology getting more Personal or Intimate?”)

I think this transition will occur in steps. Take the automotive industry for example. First, we will put cameras in to allow drivers to make better decisions. For instance, a camera can look at a driver’s blind spot and pass along the data of whether a car is there or not. The driver makes a decision about what to do with that data. In the second iteration of the technology, the camera becomes the second opinion on a decision. If I try to change lanes and the camera sees a car in the other lane, it tightens the tension in the steering wheel to let me know there is a potential problem. Third, the camera becomes the leading decision maker and the driver has to option to override it. Ultimately, we will reach a point where the camera is the only opinion and decision maker. (“Is There a Consumer Breaking Point In Terms Of Convergence?” )

Of course, all of this is going to require programmability. Not only because the system has to be instructed in the myriad of decisions it will have to make, but also as a reality of business. We will need cameras that are programmable and able to have the software adapted according to the application, which will continue to evolve with time. But the bottom line is, as long as you have enough programmability, there’s no limit to what you can do. (SoCs Are Answering Demand for Converged and “Architecture or Application?”)

Of course, we humans will need some re-programming as well. We have to be able to trust these cameras and the decisions they make.

But once we do, we will see these smart cameras used in every aspect f our daily lives. They will be used by the department of transportation to detect not only the location of traffic problems, but the root cause. They will be used for every form of security, from facial recognition in airports, to guarding newborn infants in hospitals. They will be used to transform transportation as computers relieve us of the burden of driving, thus ending gridlock, accidents and delays.

The technology will be in place. Are we ready for it as human beings?

This entry may also be found on SemiApps.com

Jeremiah Golston
Chief Technical Officer, TI Streaming Media

Let’s pick up our discussion on HD where we left off several weeks ago. In these blog posts, we talked a lot about the digital video revolution that is transforming home entertainment and information systems. What we haven’t talked much about are the complex issues involving content interchange that are inherent in such a revolution.

HDTV is now securely placed at the upper end of home video, and video content suppliers are migrating to advanced codec algorithms such as H.264/MPEG-4 in order to supply more programming with the same bandwidth and disk space. But the need continues for support of legacy codecs (think MPEG-2). Also, a growing number of networked devices are being produced that can playback video both in and away from the home. Featuring a wide variety of formats and bandwidth capabilities, these equipments need to be able to share content, including HD video, so that the connected home doesn’t become a virtual Tower of Babel.

These factors all point to the need for transcoding: converting codec formats, resolutions, frame rates and bitrates to allow an original audio-video content source to be played back on any number of different gadgets with different capabilities. Though transcoding has long been familiar in network infrastructure equipment, its importance is only beginning to be felt in consumer devices.

To provide HD transcoding capabilities in cost-sensitive mass market video appliances, system developers will have to rethink their designs. Greater signal processing performance, system memory and I/O will all be necessary, as well as programming flexibility in order to support various codec requirements with the same hardware design. The processor used will have to provide a platform with high video performance that scales up to HD data rates, plus the right balance of on-chip memory and peripherals for affordable video design.

In the next few blogs, I want to take a deeper look at why transcoding is a vital piece of the HD puzzle. Hopefully, I won’t babble on.

Jeremiah Golston
Chief Technical Officer, TI Streaming Media

This past week I had my annual trek to Amsterdam for the IBC show, which has become one of my favorite trade shows (except for the fact that it runs for five straight days including the full weekend). It is a great place to meet customers and key industry technologists across the entire ecosystem from production, content management, head-end encoders, servers, conditional access and client set-top boxes. One of the fun aspects is commuting to the show on the tram each day and mingling with a random mix of other players in the market. The Indonesian food is also excellent and a highlight each year!

It is interesting to watch the focus shift from one year to the next and see how the market progresses versus expectations from the previous year.

In 2003 we demonstrated support for the new advanced codecs, including H.264 and WMV9. This led to significant design-ins for the emerging IP set-top box business. In 2004 we saw early prototypes with the first full interop trials with head-end encoders, and by last year these boxes were entering production.

Much of the focus last year was on how the market would migrate to high-definition, including the concept of HD-ready boxes that would add HD support ahead of the actual capabilities of the network.

This year, while initial HD IP STBs are starting to ship, much of the focus shifted back to developing lower cost SD solutions since almost all of Asia and Latin America and much of the European market is still SD and operators have come to grips with higher costs of HD boxes versus optimized SD solutions.

Highlights in the TI booth this year included portable media players, digital media adapters, IP STBs and IP videophones, all implemented on the DaVinci™ technology platform. Early demonstrations of converged applications included a high-quality H.264 videophone at full VGA resolution integrated with a full IP STB interoperating with head-end encoders and VOD servers.

The flexibility of DaVinci technology enables multiple services to be offered on the same box, providing additional revenue opportunities for operators. Significant software development is required to integrate these capabilities seamlessly in the same product, but I’m betting that someone rises to the challenge to deliver converged products next year, building on the concepts in our early demos.

If you were at IBC and saw other interesting trends (or have a great Indonesian restaurant recommendation), we’d love to hear from you.

Jeremiah Golston
Chief Technical Officer, TI Streaming Media

Through examples like Lifesize and digital camera, we have seen that there are various reasons for choosing to implement HD. We have also seen that the right mix of flexibility and system integration is critical. But is HD always necessary?

In some cases, HD is not needed at all. Standard Definition (SD) is still the best tradeoff in applications like IPTV in Asia where BOM advantages (less memory for example) take precedence and limited network bandwidth makes delivering even high-quality SD still a challenge. Conversely, there are cases where HD is not enough. Medical imaging applications such as ultrasound require an even higher resolution than 1080i. State-of-the-art ultrasound applications require 4Kx2K to produce the detail needed to make proper diagnoses.

Manufacturers also have to consider the versatility of their designs, since virtually every digital video system designed today has to take into account the continual introduction of improved codecs. Whether for DTV broadcasts, IPTV, videoconferencing or other applications, the influence of H.264/MPEG-4 AVC in the next few years will be significant. There are competing standards, too, such as WMV9/VC-1 and China’s AVS, and the ITU/ISO standards which all allow for variations in implementation that greatly impact the quality of the resulting video.

Systems such as set-top boxes (STBs) may have to deal dynamically with any number of standards and variations; they may also have to interface to entertainment and gaming consoles, as well as support home computer networks and eventually videophones. It may be important for such a system to not only decode, but also transcode video streams to share content with other devices in the home. Even a more closed application such as video surveillance needs the ability to upgrade codecs and add features such as object analysis and recognition.

Designers have to bear all these factors in mind as they select an enabling technology for their video systems. Sometimes HD is the right way to go. Sometimes it just isn’t. Either way the future of video applications is booming and the growth is exciting.

Jeremiah Golston
Chief Technical Officer, TI Streaming Media

Based on my example in my last blog of Lifesize’s video conferencing application, it’s clear that best-in-class compression is one of the issues that has to be taken into account when implementing HD. But it is not the only one.

One of our digital camera customers used HD in its video-to-print application that didn’t require the absolute best compression ratio, but a traditional motion JPEG approach just wasn’t good enough. For them, the key was to be able to use motion video to capture a great still shot from video. Think of parents trying to get distracted children together for the perfect picture – this generally requires many attempts as I can testify. With the camera’s video application, you can see frame by frame shots and choose the best one, sending it straight to print. The key ingredient was to take advantage of the fact that the front end could already capture 720p and then a programmable video engine enabled flexible encoding that could adapt to print from individual frames.

Speaking of the front end… It is no small feat to capture images with high enough quality to print as well as view on a high definition display at home. This process, that we might take for granted, requires a high performance digital image processing pipeline capable of handling HD resolutions at video rates. Key elements in the signal processing chain include color space conversions, noise filtering, video stabilization, and collecting statistics for auto-white balancing. The camera must also generate preview images and video in real-time for display on the smaller camera LCD screen.

To support this high-throughput HD video system on a chip application requires a solution that combines performance, system integration, and versatility at a reasonable cost. Solutions using TI’s DaVinci™ technology integrate a sophisticated image processing pipeline for front-end capture, back-end display processing including on-screen display (OSD) and integrated video DACs, flash card interfaces, USB for image and video download, flexible video and audio compression engines using programmable DSP and hardware acceleration, and a host RISC processor for control, communications, and application software.

Programmable engines provide the flexibility needed to support codec customization such as the 720p print from video application and also allow the system to be scaled readily to accommodate new functionality. However, none of this would matter for this demanding application without the ability to capture crystal clear individual images with the highest quality using today’s most sophisticated image processing algorithms. With the right combination of programmable flexibility and hardware acceleration, the same basic design can be redesigned to meet requirements for different market segments and even different regions without losing the application specificity needed to meet demanding system cost requirements.

HD videoconferencing and print from video capabilities in digital still cameras are just two examples of how HD processing must be customized to enable cool new applications. At TI, we are working with a range of video customers who face unique HD processing requirements to create break-through capabilities in their products. While the exact chip architecture will vary, they need the right mix of flexibility and system integration to turn their ideas into reality.

Jeremiah Golston
Chief Technical Officer, TI Streaming Media

I’m continually surprised by the applications that are taking advantage of high definition (HD) video. The trade-offs from standard definition (SD) are negligible in some cases, and in other cases HD makes no sense at all. I would not be surprised to see even cell phones requiring some HD encoding support soon. Despite the limited screen size on the phone itself, camera phones will want to capture short high quality video clips to watch later in the home on a higher resolution display.

When start-up company LifeSize first talked to us about bringing HD quality to video conferencing a few years ago, I thought they were crazy. My opinion was the issues facing video conferencing had nothing to do with resolution but instead centered on ease-of-use and bandwidth for good SD transmission. In retrospect, they were exactly right. With HD videoconferencing, the overall experience is much improved. For one thing, meetings flow more naturally with the ability to draw on a physical white board and have the text and diagrams visible to the far end instead of having to use a separate white board application. Additionally, facial expressions are much more discernable in high definition and this is one of the real benefits for adding video on top of a standard conference call. Video conferencing room systems typically already use large screen displays – a natural fit for higher resolution if the compression limits can be addressed.

Raw 1080i60 video has six times and 720p60 more than five times the amount of data as SD video. In terms of basic processing throughput, I/O bandwidth, and memory requirements for video buffering the resolution alone can drive the system requirements up by a factor of six for HD compared to SD. Moreover, HD applications often demand the higher compression efficiency of advanced codecs like H.264. Newer advanced codecs achieve greater compression by employing even more memory and processing so the system requirements become correspondingly even higher. These requirements translate into higher component costs that will diminish over time.

This is something Lifesize took into account and designed around to great effect. They found that 720p30 was currently the best fit for their application. Almost all HDTVs today have 720p max resolution, so encoding at 1080i or 1080p would be a waste. However, I think we will see the video conferencing market move to 1080i or 720p60 over time as a way to offer lower latency. Until then, 720p30 offers the perfect tradeoff.

Normally, digital video is compressed to reduce the enormous bandwidth required, which exceeds 124 Mbps for SDTV broadcast formats and approaches 750 Mbps for 1080i60. Storage is a factor, too, since single-layer DVDs can hold about 4.7 GBytes of data—enough for only short clips of uncompressed video. Single-layer HD-DVD and Blu-ray discs extend storage to about 15 and 25 GBytes, respectively, but they still require a huge amount of compression to hold a full-length HD movie on a disc.

The Main Profile of MPEG-2, the best established and most widely used standard for video compression, can normally provide high-quality compression for difficult content at ratios of around 30:1 for SD and 50:1 for HD, using 4:2:0 color sampling and depending on the source. Since H.264/MPEG-4 AVC, High Profile, roughly doubles this level of compression, the video broadcast and recording industry will be transitioning to the new standard during the next few years. All of the ITU/MPEG standards are lossy, however, so the decompressed image played back is by nature less well defined than the original image before compression. Because the images are in motion, and because the standards are based on a great deal of study about how people perceive images, the loss of image definition is concealed so well that it is generally acceptable provided the compression ratio is not pushed too high.

However, pushing this loss beyond the ~60-100:1 compression ratio supported by H.264’s High Profile risks revealing flaws in the image, and these flaws show up much better with HD displays. If you find the network bandwidth or storage requirements for your application demands compression rates higher than these ratios to support a given flavor of HD, you are likely better off staying with a lower resolution. You may also find that HD displays are not only unnecessary, they may be distracting and even unpleasant to view.

As you can see, the considerations a designer must take into account are extensive and should be tailored to the specific application requirements. However, the applications for HD do extend far beyond the traditional display end equipments typically discussed.

Jeremiah Golston
Chief Technical Officer, TI Streaming Media

Before we dive headlong into a discussion of HD and what it means to designers today, it would be instructive to start at the beginning with a quick recap of how we got here. What has been the driver for HD?

The original push for HD came as a result of widescreen movies. Ironically, these widescreen formats were introduced by movie makers as a direct response to the threat of television in the 1950s (think Cinemascope). The movie studios wanted to offer a viewing experience that couldn’t be duplicated on the tube. Anyone who has watched at home a non-letterboxed version of movies from this era – like Ben Hur – knows that much of the picture is lost when translated to a traditional 4×3 aspect ratio. An editing method known as pan and scan has to be used to crop the images. This holds true for most movies to date not filmed in a digital format.

The first HDTV technology was also developed for movies. In the late 1970s, Sony and NHK developed NHK Hi-vision, which was able to produce images with the same quality as 35-mm film. In the 1980s, interest began to build for the development of an HDTV system for commercial broadcast that would have roughly twice the number of vertical and horizontal lines compared to conventional broadcast systems.

HD offers the ability to increase the definition per unit area and also to expand the percentage of the visual field contained by the image. The goal is to exponentially increase the number of horizontal and vertical pixels available in SD, as well as changing the aspect ratio to 16×9 from 4×3 thus making the image more movie-like. The following chart shows how pixel rates have changed relative to SD.

Obviously the push for HD goes beyond television. Today there is demand for HD in streaming media, DVDs, PVRs, security surveillance, digital still cameras, digital camcorders, cell phones and even videophones. As a result, market analysts are predicting explosive growth. For HDTV alone, we are expecting to see nearly 120 million units shipped by 2008.

So the demand is there – though Gene might argue that the technology is driving the demand without a corresponding need for such a good picture (read his anecdote about sound quality influencing video perception). Regardless, we need to discuss what this means to you as a system designer. Hopefully we can get a good discussion going about, not only the opportunities, but also the tough choices that have to be made. I have my own ideas which I look forward to sharing – but I don’t have all the answers. Agree or disagree, I’d like to hear from you.

Jeremiah Golston
Chief Technical Officer, TI Streaming Media

For then next few entries, I’ll be discussing in some detail a subject that is at the forefront of digital video design and one that is near and dear to both Gene and me: high definition (HD).

It’s funny but even though I work in video and have expertise in discerning different video artifacts, I was never too concerned about the video quality at home and didn’t even buy an HDTV until last year Unlike Gene, I’m too cheap to be an early adopter when prices are still expensive.

Once I saw how stunning the quality of our HDTV with HD programming was, I was hooked. This was, of course, after I actually connected the HD ATSC input correctly - something our installer hadn’t done because he assumed we would be upgrading our satellite receiver in the next month to support HD. This meant that we weren’t actually getting HD programming on the TV from the initial install – but I digress.

Within a week my wife didn’t want to watch or record anything that wasn’t in HD and soon we were both annoyed at having to watch standard definition (SD) programs. Initially we didn’t have an HD DVR forcing us to record programs in SD. Meanwhile, our TV viewing habits had changed over the last several years to watching almost everything off the DVR because of the convenience and being able to cut down on the viewing time by skipping commercials. Forced to choose between HD and DVR, we were selecting DVR but my wife wasn’t happy. We upgraded to an HD DVR as quickly as we could and for once we did become an early adopter. Now our main issue is that, aside from the local channels which are in HD, our satellite provider has only a limited number of HD channels. This will hopefully change over the next few years as they fully deploy the new MPEG-4 AVC codec (also known as H.264) along with new satellites using DVB-S2 to increase their HD channel capacity.

What’s the point of this story? My wife and I have become HD snobs – and if we are, soon everyone will be. HD is that big.

HD displays for DTV and other applications are an important part of the digital video revolution, and video system manufacturers are faced now more than ever with decisions about implementing HD. Over my next few blog entries, we’ll discuss many flavors of HD and the challenges of making implementation trade-offs across a range of applications outside of DTV. Perhaps the most important will be actually defining what HD is as it’s proving to be a slippery term. We will also look at the factors that affect perceived image quality, the impact of HD on system cost and flexibility, which applications will benefit from HD, how to choose a media processor that can handle this cutting-edge technology, and whether designers, themselves, are HD-ready.

I’m looking forward to receiving your thoughts on this subject over the next few weeks. At that time, I hope we can all embrace our HD snobbishness.