Welcome, Sebastian Schuster!

Welcome back to the University of Vienna and congratulations on your success in the WWTF Vienna Research Groups for Young Investigators Call 2023! What motivated you to submit a project to the WWTF Call with proponent Benjamin Roth?

Benjamin Roth established a group here that is doing really exciting and cutting-edge research in Natural Language Processing (NLP) and I found that this call was a great opportunity to join forces with him to make Vienna a center for fundamental NLP research. The support from the WWTF to set up my own group is quite amazing and provides me with the resources to keep doing my research and to train PhD students to become independent researchers themselves.

On top of that, both the city of Vienna and the University of Vienna are special to me given that this is where I grew up and where I started my academic career. I’m both excited that I get to live again in Vienna and that after 13 years abroad, I get to share my experience with the next generations of students.

Let's go back to the beginning! You completed your bachelor’s degree in computer science at the University of Vienna. What initially motivated you to start studying computer science?

I’ve always enjoyed programming and found thinking in formal frameworks quite natural, so computer science seemed like a good fit when I started university. What I’ve always really liked about computer science is that its methods play such an important role in virtually any discipline, so it’s possible to collaborate with and learn from researchers from lots of different subjects. During my bachelor’s degree, I worked a lot with bioinformaticians and biologists, and then later with researchers from psychology, sociology, and most and foremost; linguists.

What fascinates you about your current research on "Large Language Models"?

In the history of humanity, there has never been anything non-human that can produce language in an (almost) human-like way. As far as we know, there is no other animal that has a communication system that is as complex and versatile as human language, and yet we’ve created an artificial system in the form of Large Language Models (LLMs) that are able to produce responses in fluent human language to almost any prompt that a user provides. And while the general principles of how these models work are quite simple – they are a machine learning model trained to predict one word at a time based on the previous words – we still have only a very limited understanding of how these models achieve this feat at a mechanistic level.

Figuring this out is in my opinion a super fascinating topic since first of all, it helps us determine for what kind of applications these models are well suited and for which kind of applications they pose too many risks. Second, if we can fully understand how these models achieve human-like linguistic competence, we may also be able to reverse-engineer to some extent how humans process and produce language, and more generally, how the human brain works, which I similarly find a very fascinating question.

Reflecting on your academic journey, how did your studies and previous positions at different universities shape your research interests and career path?

To some extent, I think my research interests and career path were already shaped during high school. I attended a high school with a very strong language focus – all classes were taught in German and English and we also had French and Spanish classes which I think sparked my interest in language.
Then, during my bachelor’s degree, I was introduced a lot to statistical and machine learning models and algorithms through the bioinformatics classes. This built my foundation for these kinds of models, which I still engage with on an almost daily basis in my research.
Then, during my master’s at Stanford, I was introduced to NLP and linguistics, which brought together my interests in language and in computational models.
Then, through my PhD, I was able to learn much more about these topics and I also got the chance to dabble more in the cognitive sciences and learn a lot about how humans process language and how to do research with human subjects, which has been an invaluable tool since then as well.
And then finally, in my postdocs, which coincided with the rise of Large Language Models, I was able to gain a deeper understanding of these models and develop principles of how to evaluate and study these kinds of models.

Apart from opportunities to learn about all these topics, my career has also been tremendously shaped by fantastic mentors. From researchers at CIBIV here in Vienna who taught me the basics of doing research to my advisors at Stanford and later on my postdoc advisors, I’ve been very fortunate to have been mentored by brilliant and kind people who invested a lot of time in my development – something I hope to pay back through mentoring the next generation of researchers.

You have been in academia for quite some time now. What do you see as the positive aspects of this environment? Is there anything you would like to change?

One of the most amazing things about working in academia are the people you get to meet. I am almost constantly surrounded by people who are much smarter than me and bring in very different perspectives, so there are opportunities to learn new things virtually every day. I also appreciate the amount of freedom one has in shaping the research agenda and how to approach things.

In terms of things I would like to see changed, I think there should be more investment in young researchers to establish their own independent research groups and less focus on “bean counting” when it comes to evaluating merit. Doing impactful research requires time and many failed attempts and it seems to be becoming increasingly difficult to forge a career without constantly publishing.

Thinking about your daily work, what tasks do you enjoy the most, and which ones less so?

I really enjoy research discussions with colleagues and mentees, and most of the time, I also enjoy teaching, especially if the class is on topics that are closely related to my research. Tasks I enjoy less are dealing with forms or archaic systems to get things done that would be a lot easier in smaller organizations.

Could you imagine a position outside of academia? If so, what kind of job would it be?

During my PhD I’ve always joked that I’ll open a bakery if my academic career doesn’t work out, so maybe that? I am also quite interested in design and architecture, so a job like interior designer also seems sometimes appealing. More realistically, though, I would probably work as a software engineer or project manager, if I didn’t have my current position.

Why do you think there are still fewer women in computer science than men, and what can we do about it?

I don’t think there is a single reason for this but I think it is a combination of not getting enough women to start a career in computer science and not retaining enough women in the long term. While I think the image of computer scientists has fortunately shifted and has become more appealing to a broader demographic, there are still too few female role models, which sometimes may make it more difficult for women choosing their degree program or deciding whether to continue in the field to envision themselves being part of the field in the long run. The bigger issue, though, I think is that in some areas in computer science, there still exists a bit of a macho culture that is very off-putting to many people (including me) and drives people out of the field.

I won’t pretend to have all the answers on how to solve this but what I aim to do is to actively create environments in which everyone feels welcome and make gender balance a priority in recruiting, in the hope that this will lead to more future role models and break some of these cycles.

When you think back to your time as a student, which courses did you enjoy the most? What is important to you in your role as a teacher?

I really enjoyed the bioinformatics courses back then. They were very well taught and had a strong connection to current research and we were quite lucky that most of classes had very few people in them, so we got to interact a lot with the teaching staff.

In my teaching, I therefore also aim to make courses as interactive as possible, though this can sometimes be a challenge for bigger courses. When teaching technical concepts, I also aim for high levels of consistency so that students can really focus on understanding the core of the issue rather than having to deal with varying notations from various sources. I also try to provide learning materials for different types of learners, so that they can learn the material independent of say whether they prefer hearing me explain a topic or prefer reading about it.

What advice would you give to today’s first-year students?

Interact with as many people as possible. You could learn a lot of the things taught in the courses also through other means but what makes being at a university really special is that you are surrounded by peers and teaching staff that have similar interests as you. You will gain a much deeper understanding and broader perspectives when you talk to the people around you. Also: Enroll in courses from other departments that interest you even if you don’t see any clear connection to your main area of study.

Thank you for taking your time to do this interview!