Thomas Assesses Building Damage from Natural Disasters Using Computer Vision and Interpretable Deep Learning Models

Thomas is a student from the United States who attended the Summer STEM Institute (SSI). Thomas is interested in a wide range of areas, from ecology and political science to computer science and machine learning. Over the summer, Thomas worked on a project, “Interpretability And Convolutional Neural Networks for Building Damage Assessment,” to build tools using computer vision to assess the damage on buildings before and after natural disasters. His paper was submitted and accepted into the Neural Information Processing Systems (NeurIPS) Conference, a premier conference for machine learning in AI. You can read his paper here and learn more about his research below. 

Let's start off by talking a little bit about your background and interests. How did you first become interested in science and research?

I've been interested in science since I was in elementary school. I realized that I really like how science can explain what humans have been wondering about since the beginning of time. In terms of research, I honestly didn't know much about it at all before going into SSI. However, SSI has definitely brought me into the world of research, and I ended up really enjoying it. 

In terms of the specific areas of science that I’m interested in, I’ve always enjoyed learning about ecology and wildlife. Since I was little, I loved going to zoos and watching nature documentaries. More recently, I'm actually starting another data science research project studying invasive species. 

What aspects of science or research are you currently most interested in? 

My current interests lie in machine learning and computer vision, which is what my research project in SSI was about. I’ve also taken programming classes in school and outside of school before participating in SSI.

I’m really interested in the rapidly changing state of machine learning and computer vision. When you go to these conferences, you can see the new ways to label data and new models which are coming out so frequently. For example, deep learning only came out a few years ago, and it has already spiraled into so many different forms and applications.

What was your programming experience prior to SSI?

In high school, I took AP Computer Science. I think SSI definitely condensed a lot of what I learned from that class into just two weeks, which I thought was really impressive how the program was able to do that. Outside of school, I've taken classes on the Art of Problem Solving (AoPS). That’s where I learned more in-depth programming in Python. However, because SSI is focused on programming for the purpose of conducting data science research, it was pretty different from my previous experiences. My past experiences in programming definitely helped me over the summer though because I came in with some familiarity on all the syntax and basic concepts. 

Outside of your programming classes, what other activities do you enjoy participating in?

I’m really involved in speech and debate. I enjoy debate and public speaking, and I think they’re both really important skills to develop. These skills even come in handy even when it comes to science research presentations because it’s really important to know how to convey your ideas effectively in not just a written, but also an oral medium. Apart from speech and debate, I’m also involved with Future Business Leaders of America (FBLA), and I participate in competitions. 

In terms of STEM activities, I’ve conducted research in solar terrestrial physics for two summers under a professor at a nearby university who’s affiliated with the Center for Solar Terrestrial Research. I worked on projects related to mini filament eruptions and other events that occur in the sun. The project is related to finding the correlation between mini filament eruptions and coronal jatha, or coronal jets on the sun. These are just two different kinds of solar events that occur, and in order to find the correlation between them, I analyzed data collected from the Big Bear Solar Observatory in California using computer vision and convolutional neural networks. 

Could you tell us about the research you worked on over the summer at SSI in layman's terms? 

The title of my project is “Interpretability And Convolutional Neural Networks for Building Damage Assessment.” In summary, I worked to assess building damage using deep learning and computer vision. Obviously, natural disasters are devastating, and it’s critical that we allocate resources as efficiently as possible. In order to allocate resources more efficiently, we need to be able to assess exactly which buildings were damaged and where the highest levels of damage actually have occurred. Therefore, the goal of our research is to essentially assess the damage on buildings before and after a natural disaster. We created gradient class activation maps to show where the most damage occurred.

What was the inspiration for this project?

I couldn't have come up with this idea with my mentor from SSI. His name is Ethan Weber, and he’s a graduate student from MIT who specializes in computer vision. He’d been working on a project relating to detecting damage in images. I didn't necessarily go into SSI with a very specific research area that I really wanted to explore. Since I ended up working with Ethan, I decided to go with something that he knew a lot about and that I was excited about. I thought the implications of this project were not only interesting, but also impactful and could help many people. 

What was the most valuable lesson you learned through completing this project?

I think one of the most important lessons I took away from this experience was that research is not just a straight-forward process; rather, it involves a lot of trial and error. There are many instances where things just don't work out, which can result in a lot of rethinking and changing course. For example, programming in general involves a lot of debugging. Debugging usually takes much longer than expected, especially when you find out there’s something wrong with the model or your data. Finally, I also learned a lot about the specific field of computer vision for building damage assessment just through reading the existing literature and synthesizing that information, culminating in me writing my own paper for the project.

What was the most challenging part of conducting your research project? 

Well, I had to start from scratch because I really didn’t know much about computer vision, machine learning, or data science at the beginning of the summer. It was challenging to learn everything while programming and debugging at the same time. My project was also pretty comprehensive because it involved analyzing close to hundred of thousands of images. Even though working through my research was very challenging, it was also extremely rewarding. I also felt supported throughout the whole process because my mentor was there to guide me through any bottlenecks that I had in the process. Overall, my experience was great.

On the flip side, what was your favorite part of the research process? Did you learn anything unexpected or interesting?

Seeing the results was definitely really rewarding. My favorite part was arriving at the results and seeing how they confirmed my hypothesis about which kinds of inputs were most important for the convolutional neural network to use in order to analyze a building image. I also enjoyed creating qualitative examples in my paper. 

The most unexpected thing that I learned throughout the process was that, like I said, research isn't a straight-forward process. It requires a lot of redoing and trial and error, but is ultimately a very rewarding experience. On another note, I’m really happy that I was able to build such a meaningful relationship with my mentor. We still keep in touch today, and it was really nice talking to him every day over the summer not only about research, but also other pieces of advice he had for me as well.

It's amazing to hear that you and your mentor became so close. Have you continued to work on your research after SSI?

Yes. I submitted my project to the Neural Information Processing Systems (NeurIPS) Conference, which is one of the premier conferences for machine learning in AI. The project actually got accepted in December, and I will be presenting it at the conference! Aside from that, I'm also working with my mentor and his group at his lab at MIT on two more projects. One of them relates to building off of a data set that he recently released. The other involves conducting some additional analysis on the existing project as well as a longer term project, which generally relates to damage assessment.

Oh, wow, that's amazing - congratulations! You must be busy with all of these projects, but have you been up to anything else that you’d like to share?

Aside from computer vision in building damage assessment, I'm also really interested in politics and political science. I've been doing some preliminary research on the side about politics, specifically elections, and data related to voter turnout, ad spending, and campaign finance. I want to see where I can apply machine learning to this data, and I'm definitely thinking about working on a project like this. I'm not sure if I necessarily have time for that right now, but it’s definitely a long term goal of mine because I'm interested in both politics and data science. This project could be a really interesting way to combine these interests. 

How has working on your research project over the summer influenced your interests or plans for the future?

It has definitely influenced my plans and my trajectory in high school, college and beyond. SSI turned out to be a great opportunity, and I learned so much about research. Before this experience, I wasn't very knowledgeable about data science at all. Now I realize that research isn’t just conducted in a traditional wet lab. This was key to changing my perspective. Because of this experience amongst others, I know that I want to continue conducting research in college, and possibly even beyond that!