Takanishi Lab

After finishing my master’s degree in Germany I went to work in Shanghai for a couple of years. One day I did a tour of the computer science department at Fudan University. Fudan University at that time was famous for its Mathematics department and generally had the reputation of being a great university.

A young computer scientist named Xianyang Xue gave me a tour of the computer science department. While Xianyang’s focus was primarily on video analysis he also showed me a humanoid robot a colleague had been working on for some time. Compared to the humanoid robots you see in China nowadays this robot didn’t seem specifically advanced, still, it looked great and for some reason this triggered in me the idea that I’d want to work on humanoid robots.

At that time humanoid robots were a very niche research topic. The only way to work on such a project full-time was to pursue a PhD. There were two well known research clusters for humanoid robotics, one was at MIT where Rodney Brooks had pioneered humanoid robotics research since the 80s. His student Cynthia Breazeal had specialized on behavioral, human-interactive robotics.

The other cluster was the humanoid robotics institute (HRI) at Waseda University in Tokyo. The lab at Waseda University was originally founded by Prof. Kato who famously had built an anthropomorphic piano playing robot that was exhibited at a world exhibition in Tsukuba in 1985. Prof. Atsuo Takanishi had then taken over the lab from Prof. Kato and continued his work on humanoids. Takanishi Lab had been exploring very similar topics to Cynthia Breazeal and had expanded significantly on the topic of expressive robotic interaction. In fact their Waseda Eye 4R robot had several features that the team at MIT hadn’t been able to implement with Kismet yet, such as an emotional cortex, audio in- and output as well as arms and a torso. Very impressive!

What eventually brought me to the conclusion to pursue studying in Tokyo was actually a German Donald Duck comic book I had read in my childhood. That piano playing robot that Prof. Kato had presented at the world expo had been featured in this comic book and I still remembered it. I felt I was going “full circle”!

At Takanishi lab I started working on the Waseda Flutist robot, an anthropomorphic robot emulating human flute playing. The focus of this project was less to build a robot that can excel at playing an instrument like a human, but rather the challenge of artificially recreating the complex mechanics involved in creating the flute sound using human-like motor skills.

At the time I joined the lab the work on the flutist robot had been going on for several generations of students and robot versions. Therefore the mechanics and control of the robot were already in a very advanced stage.

It was interesting to experience how the distribution of roles between the student generations worked. Bachelor students would usually work on the mechanical design of parts of the robot, while master students worked on the more scientific, usually software driven problems. Passing the knowledge from one generation of students to the next would be almost like a ritual, following the traditional Japanese principle of 先輩 (older student / teacher) and 後輩 (younger student / learner).

For my research I kept the focus on my original interest of exploring the interaction between robot and human. In case of the flutist robot I wanted to find out how the two could work together musically. The fundamental idea of my PhD thesis was therefore that a human and a robot playing a musical piece together would deliver a more engaging and entertaining performance, if they had means of interacting with each other while playing a song.

At least in my theory, having a robot just plainly accompany a human while playing, would result in a static unnatural performance that would be less entertaining to a human audience. On the contrary, as soon as some kind of interaction took place, the performance would be lively, human-like and much more engaging.

I took this idea from the fact that Jazz musicians in a band would use multiple ways to give subtle cues to each other, either by playing certain musical patterns, or by moving their instruments in a certain way.

While I did explore both, the aural and the visual interaction strategies between musicians, I initially worked on allowing the robot to understand visual feedback from other musicians. While I describe several approaches to do this in my thesis, a relatively simple method turned out to be quite effective. I called it the virtual fader. The virtual fader, like its mechanical counterpart, consists of a bar that can be moved back and forth, but not by moving your finger but by moving an object in the visual space of the robot. In this way I was able to manipulate live performance parameters of the robot by just moving my saxophone back and forth.

A PhD thesis in Japan usually takes around 4 years, and my time at Takanishi Lab passed quite quickly. While the work on the interaction system progressed quite well, one of the hardest challenges was the robot itself. The flutist robot was a very complex device with more than 50 degrees of freedom. If only one finger didn’t move correctly or the shape of the robot’s lips wouldn’t be shaped adequately, an experiment would fail. To get the whole system to a point where its performance was robust enough to focus more on interaction with humans took a lot of time and effort. But, in the end it all worked out great!

In summary, my core assumption that an interactive human-robot performance would be more entertaining than a passive playback, including the technical methodology behind it, proved to be correct. If you’d like to have a look at the details, download my thesis from here.

I’m excited that humanoid robots are becoming prime time news now and am so happy to be able to finally engage deeper into this field with a commercially viable perspective. In a way the recent progress in AI and humanoids gives new legitimacy to my research and also to a lot of other research done at Takanishi Lab. Back in the day exploring the interaction between robots and humans within a human environment would be looked at as distant science fiction. Turns out these ideas weren’t so far-fetched at all.

There are a lot more episodes about my time at Takanishi Lab, stay tuned! Thank you to Prof. Takanishi and Prof. Jorge Solis for their wonderful support and endless patience with me :)

Petersen Instruments Part 2 - The Magnetodrome

At a similar time as the VFTB my dad and me started working on the Bacteriodrome as shown below, or later, the Magnetodrome. These instruments have two application areas.

In biomagnetism magnetotactic bacteria are organisms that navigate based on the earth’s magnetic field. They do this in lakes and lake sediments to find nutrients located in certain areas at the bottom of a lake. This navigation is possible because the bacteria have small magnetite particles built-in that act as a sensor for the environment magnetic field.

To explore the behavior of these magnetotactic bacteria, one of most prominent is the Magnetobacterium Bavaricum, it makes sense to generate magnetic fields of variable strength and direction. A typical example is a rotating field that makes the bacteria move in a circle. Using image processing the trace of this rotation can be plotted and used for characterization of the magnetotactic organism. My father discovered the first Magnetobacterium Bavaricum in lake Chiemsee in Bavaria, Southern Germany.

The image below shows an example of the bacteriodrome plotting the traces of magnetic bacteria swimming in a rotating magnetic field. These images are created by adding the difference between two intensity thresholded adjancent video frames.

The more modern version of the bacteriodrome is the magnetodrome. Instead of the simple Helmholtz coils of the original design the system uses powerful electromagnets to move magnetotactic bacteria, but also other magnetic samples such as environmental particles or even nano-robots. One can think of the magnetometer as a set of magnetic tweezers to manipulate magnetic particles of all kinds.

As with the VFTB my primary work on the magnetodrome was the development of its control software. While overall the control of the system was a lot less complex than the VFTB, the key part here was the image processing of the camera stream from the microscope. While at some point we ported all image processing to work with OpenCV and Nvidia CUDA in high resolution, we initially used digitizers and video processing libraries by Matrox that had started working in this field back in the 1970’s.

Similarly to the work on the VFTB, Mag-Instruments has done a fantastic job modernizing the magnetodrome with a redesign of the mechnical design of the system as well as the control and image processing software. The photo below shows this re-newed version.

Petersen Instruments Part 1 - The VFTB

I launched the first series of devices that became products while I was still going to Highschool in Munich, Germany. The idea and overall concept for these devices came from my dad who’s a professor at LMU Munich. As a geophysicist he’s interested in the magnetic characteristics of geological and biological samples. For the purpose of measuring the magnetism of such samples we developed two devices together, the variable field translation balance or VFTB, and the bacteriodrome that later became the magnetodrome.

The variable field translation balance (VFTB) suspends a rock sample on a miniature swing and then actuates the sample using a magnetic gradient field. The sample is magnetized by an superimposed stronger electromagnetic field. The response motion of the sample to a sinusoidal actuation is measured and from it the amount of magnetization calculated.

Building this device together with my dad as a teenager was a very educational but also super fun experience. The photo above shows the VFTB we installed at the University of Hawaii Manoa. My dad and his friend Bill, a brilliant electronics engineer, developed the hardware parts of the VFTB, whereas I wrote the software to control the device. These were the times of MS-DOS and I developed the first version of the control software in a C++ dialect called Turbo C++, created by a company named Borland. While the company Borland itself was acquired and the name eventually abandoned as far as I know, interestingly, Turbo C++ still is still around in a incarnation called C++ Builder . See below a schematic overview of the different components of the measurement system itself.

Even though Windows and other graphical user interfaces like Digital Research’s GEM for the DOS platform already existed, they weren’t very popular yet. Therefore one of the most fun parts about the initial stages of the VFTB product development were that I’d have to code the graphical user interface under DOS by myself, if we didn’t want customers to operate the device from the command line. This meant frequent sessions with my dad or his students at the lab to get feedback about the user experience and making the whole application useful. The result was very rudimentary in terms of GUI functionality, but a useful, simple interface for controlling the system.

As the application needed to operate the various sub-devices of the instrument, this required communication with hardware such as the temperature controller via various interfaces, mainly the multiple COM ports of the control PC. The system also included an A/D converter card built into the computer as 16-bit ISA extension card.

Below is a screenshot of the original VTFB Control application running in a DOSBox emulator. Unfortunately, without access to one of the old machines, I don’t have any images of this software running an actual measurement.

After a while we were installing VFTBs all over the world. Not in huge numbers, building the VFTB was never about commercial success, it was about providing a uniquely sophisticated measurement device to the research community. Some of the first systems went to the UK, to Oxford University, to Liverpool University and to the University of Southampton. Key for the sale and distribution of the systems was my dad’s colleague, a professor at Liverpool university who founded his own business Magnetic Measurements to build and distribute devices in the field of Geophysics. In the coming years we made it all the way to Australia, India, China and Hawaii. It was fantastic! Apart from the technological challenge working and travelling with my dad, occasionally my mom would also join, was a wonderful family experience.

A few years ago my friend Przemek, an excelllent robotics engineer who also graduated from Waseda University in Japan, became interested in the technology of the VFTB and started a new company in Munich, Mag-Instruments, to further develop, manufacture and sell the machine my dad and me had created. Przemek turned our, in some ways still prototype, device into a state-of-the-art modern machine. The photos below show the journey from one of the first product versions we made, installed at GFZ in Potsdam, Germany, to the polished version that Mag-Instruments is selling now. What a transformation!